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[57] ABSTRACT 

For use in composing and/or editing a multimedia work that 
include one or more clips, a technique for indicating an 
active state of die clip or portion thereof, indicating effects 
applied to a track or a clip, and indicating a plurality of 
effects applied to a track or to a clip. At least a portion of the 
work is graphically represented by one or more timelines on 
which are disposed one or more clips. An active portion of 
a clip is indicated by a colored bar that extends under that 
portion of the clip. Different types of transitions between 
clips are indicated by using one of different predetermined 
colors for the colored bar. Effects applied to a track are also 
represented by color bars that extend above the portion of 
the track to which the effects apply and are independent of 
changes to any clip disposed on the track. Predetermined 
colors are used to indicate the type or class of effects) 
applied. A plurality of effects applied to a clip or to a track 
are indicated by stacking the colored bars. The duration of 
the effect(s) are indicated by the relative length of these 
colored bars. 

46 Claims, 11 Drawing Sheets 
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INDICATING ACTIVENESS OF CLIPS AND 
APPLYING EFFECTS TO CLIPS AND 
TRACKS IN A TIMELINE OF A 
MULTIMEDIA WORK 

5 

FIELD OF THE INVENTION 

This invention generally relates to a system for creating 
and modifying a multimedia work, and more specifically, to 
a system and method for indicating clips that are active and 
for indicating effects that are applied to components of the 10 
multimedia work. 

BACKGROUND OF THE INVENTION 

When creating a multimedia work that may include a 
plurality of different components such as ariirnations. video 15 
clips, and audio clips, authoring software typically provides 
a multitude of options for modifying various aspects of the 
components comprising the work. The components of a 
graphics sequence created on a computer may comprise a 
plurality of clips, each of which typically comprises a 20 
plurality of frames. When displayed at a sufficiently fast rate, 
the objects shown in different positions or attitudes in 
successive frames appear to move, creating an animation. A 
similar approach is used in video clips, except that the 
frames of a video clip comprise digitized images rather than 25 
graphic illustrations such as those that are typically pro- 
duced by a two or three-dimensional paint program, 

A multimedia work is created by combining various clips. 
It is generally desirable to provide the user with an indica- ^ 
tion of the state of the clips loaded into the authoring 
software. Each of the prior art multimedia authoring soft- 
ware products uses a slightly different user interface to show 
a track and to indicate the state of the clips comprising a 
track. The temporal arrangement of clips is generally indi- 35 
cated by an arrangement of the clips along either a horizontal 
or a vertical axis and is referred to as a "timeline." 

For example, in Quantel's HENRY™ software program, 
the timeline is vertical arrangement of clips. Successive 
clips that run sequentially appear in the vertical timeline ^ 
with each successive frame being represented by a rectangle 
below the preceding frame in the clip. Likewise, each 
successive clip follows below the preceding clip. A color 
coded bar extends down the left side of each clip to indicate 
its state, e.g.. the bar is yellow to indicate that a clip is active 45 
and that no other portion of any other clip is active. If two 
successive clips A and B overlap where a transition occurs 
between the clips, a different color bar is provided that 
extends along the side of only those frames of the clips that 
overlap in time. ^ 

In Avid's MEDIA COMPOSER™, the timeline is hori- 
zontal Multiple clips can be shown on different tracks 
during overlapping times; however, vertical ordering of the 
clips determines the clip or part of a clip that is active at a 
given time. The program treats any portion of a clip on a 55 
track above a portion of another clip (located on a vertically 
lower track) as the active portion of the clip. Thus, the active 
portion of any clip is controlled by the relative vertical 
position of that clip compared to other clips. It will be 
apparent that the techniques used in the prior art to indicate 60 
the active clip or active portion of a clip in a non-linear 
editing system do not provide much versatility. Ideally, an 
operator should be able to see all options that are available 
for use in a work and be able to immediately perceive the 
options that are being used in the output signal. 55 

An analogous problem arises during the live broadcast of 
a sports event in which multiple cameras disposed at dif- 
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ferent locations and capturing different elements of the event 
are all potentially a source of a network feed signal that is 
broadcast (or taped). The director of the production must 
select a particular camera for the signal output at any one 
time. All of the cameras feed to a central control panel for 
display on an array of monitors so that all of the views from 
which one can be selected for the output signal are visible. 
An indicator identifies which camera is currently active. 
This approach permits great flexibility and is extremely 
efficient in enabling an optimum output signal to be selected 
as the scene being televised or taped changes. Clearly, it 
would be desirable to extend this concept to multimedia 
authoring software to enable a user to view all of the clips 
from which the active clip or portion thereof can be selected 
and to indicate which portion of any clip is active. Currently, 
such features are not available. 

Multimedia authoring systems typically provide a number 
of different effects that can be applied to components of the 
work to modify the output The term "effect" refers to any 
processing applied to input data that in some manner alters 
the data. Effects such as dissolve, wipe, and various digital 
video effect transitions can be applied to control the appear- 
ance applied to a change from one clip to another. Other 
visual effects such as blur, color correction, and chroma key 
modify the overall visual appearance of a clip or portion 
thereof. Still other effects, such as echo, are applied exclu- 
sively to audio clips. Since a user can apply any of a rather 
large number of selected effects to change the result pro- 
duced in most multimedia authoring software, it is important 
to provide a visual indication of the type or class of effect 
being applied and to clearly indicate the portion of a clip to 
which an effect will apply. The prior art interface schemes 
for indicating effects differs between each program. Color 
bars have been used to indicate that an effect is being applied 
to an entire track or to show the portion of a clip to which 
the effect applies, with the specific color indicating the type 
or class of effect being applied. However, there is no 
provision in the prior art for applying an effect to only a 
specific portion of a track that is less than the entire track so 
that the effect is independent of any clip that is placed in that 
portion. Enabling a user to associate an effect with a specific 
portion of a track rather than with a clip would have a 
significant advantage. The user would then be able to change 
the clip inserted on the track at that point, and the effect 
associated with that portion of the track would automatically 
apply to any clip that is thus inserted. 

There are clearly times when it will be desirable to apply 
multiple effects to different portions of a track or a clip. 
Currently, multimedia authoring software does not provide 
an effective procedure for selectively applying multiple 
effects to limited portions of a track. Typically, in prior art 
multimedia authoring systems, an effect is applied to the 
entire duration of a clip. If the user wants to apply a first 
effect (1) to the first half of a clip and a second effect (2) only 
to the middle one-third of the clip, it will be necessary for 
the user to divide the clip into four new clips. A-D. where 
clip A is the first third of the original clip and has only effect 
( 1) applied to it, dip B is the next l A of the original clip and 
has both effects (1) and (2) applied to it. clip C is the next 
l A of the original clip and has only effect (2) applied to it 
and clip D is the last one third of the original clip and has 
no effects applied to it It will be evident that dividing clips 
to facilitate applying effects to only a portion of a clip is 
inefficient and that a better approach is required to achieve 
the desired flexibility. 

SUMMARY OF THE INVENTION 
In accord with the present invention, a method for indi- 
cating activeness of a clip to a user composing and/or editing 
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a multimedia work that includes a plurality of clips provides 
for displaying a timeline for at least a portion of the 
multimedia work. The timeline graphically represents the 
portion of the multimedia work and graphically illustrates a 
plurality of tracks for a time interval corresponding to that 5 
represented by the timeline. The user is enabled to select a 
clip as active, indicating that the clip is to be used in the 
multimedia work. The clip selected by the user is marked as 
active with a visual indicator, so that by inspection of the 
visual indicator and a position of the clip on one of the lQ 
plurality of tracks, the user can determine a time during 
which the clip contributes to the multimedia work. 

The user is also enabled to select a portion of the clip as 
active. The portion of the cup selected as active is then 
marked with the visual indicator, so that the position of the l5 
visual indicator relative to the clip indicates the portion of 
the clip that is active. Each clip is preferably represented by 
a rectangle disposed on one of the timelines. A duration of 
the clip is indicated by either its length or its width. The 
visual indicator preferably comprises a colored bar disposed ^ 
adjacent to one edge of the rectangle; the length of the 
colored bar and its position relative to the rectangle indicates 
the duration of the portion of the dip that is active. Any 
portion of a clip added to one of the timelines that does not 
overlap any portion of another clip that is already active is ^ 
automatically made active. 

Another step in the method provides for enabling the user 
to select two clips to be simultaneously active at one time 
when either a transition or an overlay is to occur between the 
two clips. A visual indicator of the duration of the transition ^ 
or the overlay between the two clips is then provided. The 
visual indicator preferably comprises a color bar having a 
color that indicates a specific type of transition or the overlay 
between the clips. 

There are several advantages to the technique used to 35 
indicate whether a clip is active in the present invention. 
Since both active and inactive clips can be placed on the 
timeline, the user can visually perceive the contents of the 
clips that are included to facilitate selecting a clip mat will 
be active. The available options are clearly presented for the 40 
user to view, enabling selection of the clip or portion of a 
clip that will be used in the work. 

Another aspect of the present invention is directed to a 
system for indicating activeness of a clip to a user compos- 
ing and/or editing a multimedia work mat includes a plu- 45 
rality of clips. The system includes a monitor for displaying 
the multimedia work and for displaying screens used to 
compose and edit the multimedia work. A memory is 
included for storing a plurality of machine instructions 
defining a software program employed for composing and so 
editing the multimedia work. A processor, which is coupled 
to the memory and to the monitor, executes the machine 
instructions stored in the memory. The functions imple- 
mented by the system as defined by the machine instructions 
are generally consistent with the steps of the method 55 
described above. 

Vet another aspect of the present invention relates to a 
method for indicating effects applied to a specific portion of 
a multimedia work when composing and/or editing the 
multimedia work. The method includes the step of enabling 60 
the user to select an effect that will apply to the multimedia 
work and to select a time interval during which the effect 
will be applied The effect is associated with a selected track 
of the timeline by the user and is applied to only a selected 
portion of the track. An effect indicator indicates the portion 63 
of the track to which the effect applies and the time interval 
of the effect 



4 

The track preferably includes an effect region that extends 
adjacent to the track. The effect indicator is disposed within 
the track region. In the preferred form of the invention, the 
effect indicator comprises a colored bar. A length of the 
colored bar and its position relative to the track indicates the 
time interval over which the effect is applied. The effect is 
applied to a portion of any clip positioned on the track with 
which the effect is associated, and which is adjacent the 
colored bar, but the effect is otherwise independent of any 
clip. 

The method enables the user to modify the time interval 
over which the effect is applied. The modification may 
include changing a point in time at which the effect is 
applied and its duration. 

In addition, the user can associate a plurality of effects 
with the track. Each effect can have a user selectable time 
interval that is independent of the time interval of other 
effects associated with the track. The plurality of effects are 
represented by corresponding color bars that are disposed 
adjacent to the track with which the plurality of effects are 
associated. A length and a position of each of the color bars 
indicates the time and duration at which each is applied to 
the track. Different colors are preferably used for the color 
bars, each color representing a different type or class of 
effect 

The order in which the plurality of effects are applied to 
the track determines the result in the multimedia work. The 
color bars representing the plurality of effects are thus 
positioned relative to each other so as to indicate the order 
in which the effects are applied. 

Another aspect of the present invention is directed to a 
system for indicating effects applied to a specific portion of 
a multimedia work when composing and/or editing the 
multimedia work in which a processor executes machine 
instructions that cause the processor to implement effects 
generally as described above. 

Yet another aspect of the present invention is directed to 
a method for indicating a plurality of effects that are applied 
to clips comprising a multimedia work, when composing 
and/or editing the multimedia work. The method includes 
the step of enabling the user to select a plurality of different 
effects that will apply to a clip of the multimedia work and 
to select a portion of the clip to which each effect will be 
applied. An effect indicator is provided for each effect to 
indicate the portion of the clip to which the effect applies and 
the time interval during which the effect applies. 

The effect indicator for each clip preferably comprises a 
colored bar. a length of the bar and its relative position 
adjacent the clip indicates the portion of the clip to which the 
effect applies. The colored bars representing each of the 
plurality of effects selected by the user for application to the 
clip are stacked adjacent to the clip. An order in which the 
colored bars are stacked indicates a sequence in which the 
effects are applied to the clip to produce a result The 
plurality of effects are associated with the clip to which the 
plurality of effects apply, so that moving the dip along the 
timeline causes movement of the effects applied to the clip. 
The method also preferably includes the step of displaying 
a descriptive label when a cursor is moved over any effect 
indicator, said descriptive label identifying a type or dass of 
effect that is represented by the effect indicator. 

Soil another aspect of the invention is a system for 
indicating a plurality of effects that are applied to cups 
comprising a multimedia work, when composing and/or 
editing the multimedia work. A processor executes machines 
instructions that cause it to implement functions generally 
consistent with the steps of the corresponding method dis- 
cussed above. 
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Yet a further aspect of the invention is directed to an 
article of manufacture that includes a nonvolatile memory 
device used to store a plurality of machine instructions. The 
machine instructions generally cause a computer to imple- 
ment the functions described above. 

BRIEF DESCRIPTION OF THE DRAWING 
FIGURES 

The foregoing aspects and many of the attendant advan- 
tages of this invention will become more readily appreciated 
as the same becomes better understood by reference to the 
following detailed description, when taken in conjunction 
with the accompanying drawings, wherein: 

FIG. 1 is an isometric view of a personal computer 
suitable for implementing the present invention; 

FIG. 2 is a schematic block diagram of functional com- 
ponents included within a processor chassis for the personal 
computer of FIG. 1; 

FIG. 3 illustrates a timeline window that includes a result 
track and up to five component tracks, in which activeness 
of a portion of a clip on a track is indicated in accord with 
the present invention; 

FIG. 4 is a portion of a track from a timeline window 
showing clip activeness and an effect applied to a portion of 
a clip are indicated; 

FIG. 5 is the track of FIG. 4. showing one of the clips 
shifted to a later point along the timeline; 

FIG. 6 is a track to which a track-based effect has been 
applied; 

FIG. 7 illustrates the change in the timeline of FIG. 6 that 
occurs when the track-based effect is shifted to a later point 
in time; 

FIG. 8 illustrates an timeline in which a track-based 
effects stack is used; 

FIG. 9 illustrates a timeline in which an effects stack is 
applied to a clip; 

FIG. 10 shows a track in which the order of the effects in 
the stack applied to the clip of FIG. 9 is changed; 

FIGS. II through 16 are a flow chart showing the logical 
steps used for determining activeness of clips and portions 
of clips on timelines; and 

FIGS. 17 and 18 are a flow chart showing the logic used 
in the present invention for handling stacked effects applied 
to clips and tracks. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

With reference to FIG. 1. a generally conventional per- 
sonal computer 30 is illustrated, which is suitable for use in 
connection with practicing the present invention. 
Alternatively, a workstation may be instead be used. Per- 
sonal computer 30 includes a processor chassis 32 in which 
are mounted a floppy disk drive 34. a hard drive 36. a 
motherboard populated with appropriate integrated circuits 
(not shown), and a power supply (also not shown), as are 
generally well known to those of ordinary skill in the art A 
monitor 38 is included for displaying graphics and text 
generated by software programs mat arc run by the personal 
computer. A mouse 40 (or other pointing device) is con- 
nected to a serial port (or to a bus port) on the rear of 
processor chassis 32. and signals from mouse 40 are con- 
veyed to the motherboard to control a cursor on the display 
and to select text menu options, and graphic components 
displayed on monitor 38 by software programs executing on 
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the personal computer. In addition, a keyboard 43 is coupled 
to the motherboard for user entry of text and commands that 
affect the running of software programs executing on the 
personal computer. 

5 Personal computer 30 also optionally includes a compact 
disk-read only memory (CD-ROM) drive 47 into which a 
CD-ROM disk may be inserted so mat executable files and 
data on the disk can be read for transfer into the memory 
and/or into storage on hard drive 36 of personal computer 

io 30. 

FIG. 2 is a block diagram illustrating some of the func- 
tional components that are included within processor chassis 
32. All of the components mounted on the motherboard or 
otherwise installed inside processor chassis 32 are not 

15 illustrated, but the functional blocks shown in FIG. 2 rep- 
resent the more significant aspects of the personal computer. 
The motherboard includes a data bus 33 to which these 
functional components are electrically connected. A display 
interface 35. for example, a video card, generates signals in 

20 response to instructions executed by a CPU S3 that are 
transmitted to monitor 38 so that graphics and text are 
displayed on the monitor. A hard drive and floppy drive 
interface 37 is coupled to data bus 33 to enable bidirectional 
flow of data and instructions between data bus 33 and floppy 

23 drive 34 or hard drive 36. Software programs executed by 
CPU 53 are typically stored on either hard drive 36, or on a 
floppy disk (not shown) that is inserted into floppy drive 34. 
The software program comprising the present invention will 
likely be distributed either on such a floppy disk, on-line via 

30 the modem, or on a CD-ROM disk, and stored on hard drive 
36 for execution by CPU 53. 

A serial/mouse port 39 (representative of the two serial 
ports typically provided) is also bidirectionally coupled to 

35 data bus 33. enabling signals developed by mouse 40 to be 
conveyed through the data bus to CPU 53. Alternatively, if 
a different device such as an external modem (not shown) is 
coupled to the serial port data can be transmitted bidirec- 
tionally between the CPU and the modem. A CD-ROM 

^ interface 59 connects CD-ROM drive 47 to data bus 33. The 
CD-ROM interface may be a small computer systems inter- 
face (SCSI) type interface or other interface appropriate for 
connection to and operation of CD-ROM drive 47. 
A keyboard interface 45 receives signals from keyboard 

45 43. coupling the signals to data bus 33 for transmission to 
CPU 53. Also coupled to data bus 33 is a network interface 
50 (which may comprise an Ethernet card for coupling the 
personal computer to a local area and/or wide area network). 
Data used in connection with the present invention may be 

so stored on a server and transferred to personal computer 30 
over the network to carry out the functions described below. 

When a software program is selected by a user to be 
executed by CPU 53. the machine instructions comprising 
the program that are stored on a memory device such as a 

55 floppy disk, a CD-ROM. or on hard drive 36 are transferred 
into a memory 51 via data bus 33. Machine instructions 
comprising the software program are executed by CPU 53. 
causing it to implement functions defined by the machine 
instructions. Memory 51 includes both a nonvolatile read 

60 only memory (ROM) in which machine instructions used for 
booting up personal computer 30 are stored, and a random 
access memory (RAM) in which machine instructions and 
data are temporarily stored when executing application 
programs. The present invention will likely be included in a 

65 multimedia authoring software program tentatively called 
DIGITAL STUDIO™ to be distributed by Softimage 
Corporation, a subsidiary of Microsoft Corporation. Due to 



04/03/2004, EAST Version: 1.4.1 



5.781,188 

7 8 

the size and the number of files comprising this software and how to configure cuts and transitions between clips, 

program, it will preferably be distributed on CD-ROM. but (Optionally, a user may turn-off the visibility of inactive 

may alternatively be distributed on floppy disks. clips and other material so that the only the material con- 

With reference to FIG. 3. qJirneline window 70 that is tributing to the output is visible.) 

displayed on monitor 38 when^fljym^a, miimmnriia, BfldJn 5 when a new clip is "dropped" onto the timeline using a 

accord with the present invention is shown. Timeline win- conventional "drag and drop" technique or entered by select- 

dow 70 includes a plurality nf Pf^ft ing an option in the menu of the multimedia authoring and 

tracks 72, 76, 82. 88, 92, and 94. Track 72 mcludes a result e * ^ ^ automatically ^ tf n0 

or c omposite clip 74, which represent the output or mewos, ^ * ^ Qn ^ {s ^ iQ mc 

for theportion displayed in turielir^nai^; nowever. a 10 ^ wimin which ^ ^ . $ ^ ± ff * 

result track is not always present. Tunc extends from left to a Qn mc ^ mat - t OVffla fl ^ of a 

right in the timeline window. JEach ^c^cornprises a plurality ^ ^ cU ^ the part of the dropped clip that 

oHrames (not separately shown). Track 76 includes two . ^ ^ q £ already active portion of another clip 

clips 78 and »0. which arc not contiguous. Only a portion of. ^ Those of ordinary skill in the art will 

each of these dips is active, i.e used m result clip 74 as 15 ^e that mmi rules can readily be applied to 

explained below. TVack 82 includes two clips84 and 86 but ^ ami ^ of a clip on me timeline, to 

only a portion of clip 86 is used in the result dip Track 88 c ICt a ^ ^ is dropped may be made active over its 

indudes a single clip 90. but clip 90 is not currently used in altering the activeness of clips previously 

the result clip. active during ^ ^ 

Alone the bottom of each track extends a region 96 in 20 ^ ' * ui * j- * 

.vv 1 _ * t a ; J\ . ^ The preceding comments are not applicable to audio. In a 

which the activeness of any portion of a clip in a track is , *\ _ ° _ . . ^T. ^ . . t 

, ' j l 1 k kJ™, th- preferred form of the present invention, there are eight 

indicated by a color coded bar extending below the active *Y , „, „, Uaw l m „ nn ,™r~™n,iJ t« 

. , ' — . ^ . 1* „ „„iT™, k„ ofi channds of audio, where each mono stream corresponds to 

portion of the clip. Tnus. in track 76. a yellow bar y© extends . , _ , , ^_ . ■ *u » L 

J_ ui u. t *«uo. / - ^ a channel; a stereo track has two channels that can be active 

below most of the ^^^^j ^St^SSL £ in Parallel and are thus not mutually exdusive. as is the case 

portion under which the ; yeUow ta ^extends is active^ and ^ ^ number of chann ^ ^ UmitC(J tQ ^ so 
induded in result cup 74. (Note that o^er color coding 

protocols can alternativdy be employed besides those used uim . J7 . . . 

in this preferred embodiment) A yellow bar 100 also **G. 3, dropping cup 84 in the position shown on the 

extends under a portion of clip 80, indicating the portion of timdiiic will automatically cause the entire clip to become 

clip 80 that is active and induded in result clip 74. In track 30 acUve « mcUcated b * y cUow bar 102 ' smcc 84 does DOt 

82, a yellow bar 102 extends under the entire length overia P ^ P 0 * 011 of ™ otha <*V 15 acUvc * 

(duration) of elm 84 in region 96. indicating mat the entire However, when clip 90 was then dropped onto the timeline, 

^nt.nr of din IU k iis^d in result clip 74. It should be noted, none of it automatically became active, since its entire 

that since a timeline c an thpnrp^llv he of infinite duration^ ^on overlaps other active portions of clips previouslyon 
a bar appearing more like a point may be appropriate to. 35 the timeline. 

represent the duration of a short clip on the timeline. The user can readily change the activeness of any dip or 

depending on the time resolution applied. A yellow bar 104 portion thereof using either options presented on a screen 

extends under most of clip 86, similarly indicating the menu by the software program or more simply by selecting 

portion of mat clip, which appears in the result clip. the activeness bar with a screen cursor (using mouse 40 or 

Part of the activeness bar under cup 78 overlaps with part 40 other P°i ntin g <k v "*) lagging the activeness bar to 

of the activeness bar under clip 86. The portions of each of «^nd or shorten its duration. The activeness bars cannot be 

the two activeness bars that overlap correspond to the dragged or otherwise extended beyond an end of a clip under 

duration of a transition between the two overlapping dips. which they lie, but they can be ddeted from a dip or added 

The transitions are represented by different colored (e.g., afl existing clip if no overlap with another active clip is 

orange) bar 106 under the right end of clip 78 and by an 45 ^ action - Transitions are created by selecting an 

identically colored (orange) bar 108 under a portion (toward edit P™ 1 <* c cut ) on a clip and changing it to a transition, 

the left end) of clip 86. Generally, except at transitions, only e.g.. by sdecting a 44 transition button" on a toolbar. This 

one clip or portion thereof is indicated as being active at one action rcsults in °n e or more transition dialog boxes opening 

time on the timeline. (This rule applies to non-audio clips; °* *e monitor, in which the user can select the type of 
however, multiple audio clips may be active at one time. The 50 transition to be applied and other parameters relating thereto, 

multiple audio clips are mixed together during the time of Although not shown, it is possible to have transitions 

their overlap. A further distinction applies to synchronized between clips disposed on the same track, 

audio clips. Any audio clip that is synchronized to a video Turning now to FIG. 4, a track 120 is illustrated in which 

clip is active only during the time that its associated video two clips 122 and 124 are induded. A yellow bar 126 
clip is active.) 55 extends under the entire length of clip 122. and a yellow bar 

During a visual transition such as a cross-fade or dissolve, 128 extends under approximately the left half of cup 124, 

two clips are concurrently active. Specific colors are pref- indicating that all of dip 122 is active, and about the first 

erably used to indicate each different type of transition, e.g.. naif of clip 124 is active. In addition, an effects bar 150 

orange for a dissolve, green for a wipe, and blue for a digital extends over the top left half of clip 124. Effects bar 130 is 
video effect Even though a cup is not active, such as clip 90. 60 preferably color coded to represent a particular effect 

it can still appear on the timeline, enabling a user composing applied to this portion of clip 124. The effect can be any of 

or editing the work to more readily dedde whether any a number of different types of effects as are commonly 

portion of the dips on the timeline should be used in the applied in digital editing programs, such as blur, 

result clip. By enabling the user to visually see all of the convolution, color correction, etc. 
clips on the timeline, including those that are not active and 65 Effect 130 is cup-based and is assodated with a single 

thus not induded in the result the user can much more clip. In FIG. 5. a track 120* is shown, which is identical to 

efficiently dedde which clips or portions of dips to indude track 120. except that in track 120\ clip 124 has been shifted 
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Co a later point in time (along with its activeness bar). When In FIGS. 9 and 10, clip-based effect stacks are illustrated 

clip 124 is moved along the timeline in this manner, effect comprising an effect bar 162 and an effect bar 164. Each of 

barJ LME is a utomatically moved with die clip. T he le ngth of . these effect bars represents a different effect that is associ- 

eff ect bar iW oeierrnines ihe portion oi cilp ii4 to which the ated with clip 142. The duration of each of the clip-based 

eff ect applies. Using the screen cursor or specified keyboar d 3 effects represented by effect bars 162 and 164 is indicated by 

shoricuts; a user can len'gtnen'or snorten tKeeBect duratio n the relative length of the effect bars. The stacking order 

and can shift the ettect to apply T lQ_anv_portiQn-oLclifia24., determines the order in which the effects are applied to the 

HoweverTthe effecTcannot extend beyond the beginning or - clip with which they are associated, with the effect bar 

end of a clip with which it is associated and is automatically closest to the clip being applied first. Thus, in FIG. 9. the 
removed if the clip is removed from the track. 1Q effect represented by effect bar 162 is applied first to the 

In contrast to the preceding example illustrating the use of underlying portion of clip 142. Since effect bar 164 is 

dip-based effects, FIGS. 6 and 7 show a track-based effect shorter, the effect it represents is applied only to the right 

applied to a track 140 Track 140 includes clips 144 and 142. portion of the first half of clip 142 as changed by the effect 

As indicated by a yellow bar 146 extending ; under clip 144 rcprc5cntcd by effect bar 162. In FIG. 10, the order of effect 

m \i^in^ff^ TJf 15 ban 162 *> d «« the processing to achieve the effects 

1^ ^ ' 15 cach * ^ result achiev ed from 

active. Along the top or track 140 is a track effects region * ~* * i j . 

150. Within me track effects region is an effect bar 152 ttiat £ S T^^if- ?f? ^^"1*5* 

extends over parts of clips 142 and 144. Again, effect bar ^fte result obtamed for the chivbased effect stack shown 

152 can represent virtually any of a multitude of different 1D G * 

effects commonly applied to portions of a multimedia work, w 10 HQS. 11 through 16, the logic flow for implementing 

If clips 142 and 144 are audio clips, the effect bar will selection and activeness in the present invention is illus- 

represent an effect appropriately applied to an audio clip, trated. Referring now to FIG. 11, in a start block 200, the 

such as echo, compression, or equalization. If the clips are program begins by monitoring input devices for multimedia 

video or three dimensional paint clips, the effect will be authoring commands provided by a user. Once a command 

visual in nature, as appropriate for that type of clip. The 25 OT action has been received, the logic flow advances to a 

duration of the effect represented by effect bar 152 is decision block 202, where the program determines if user 

indicated by the relative length of the effect bar. The type or has dropped a clip on a timeline. If the action is "a dropped 

class of effect is indicated by a predetermined color coding clip," then the logic flow advances to a logic block 204 and 

scheme, generally as noted above in connection with clip- the program makes the clip active. A clip "dropped" onto the 

based effects. It is also contemplated that other types of 30 timeline using a conventional "drag and drop" technique (or 

visual effect indicators, such as graphic icons, could be used entered by selecting an option in the menu of the program) 

to indicate each type or class of effect. Further, such other automatically becomes active if no clip (or portion thereof) 

types of visual indicators can be grouped other than in already on the timeline is currently active within the time 

vertical stacks. For example, if a timeline is vertical, the interval in which the clip is dropped. If dropped at a point 

group of icons or other visual indicators could be arrayed 35 on a timeline that overlaps a portion of a currently active 

horizontally adjacent to the timeline. clip, only the part of the dropped clip that does not overlap 

Effect bar 152 is associated with track 140 and is thus the already active portion of another clip becomes active. In 

independent of the dips placed on the track or of their a logic block 206. the program displays a color coded bar for 

location or duration. As shown in FIG. 7, the track-based that portion of the dropped clip that is now active. 

effect bar can be moved along the timeline independent of 4 0 Next, the logic flow advances to a logic block 208 and 

any clip on the track. If clip 142 is removed, effect bar 152 displays edit points at each end of the color coded bar 

is not cha nged. Any portion of a clip placed on track 140 applied to the dropped clip. After displaying the edit points, 

under effecTSar 152 will have the effect represented_bv_the, the logic flow advances to an end block 242. where the 

e ffect bar applie d ^^ ^ t p^ pJ} f gj^„pJ} p' Because effect program flow loops back to start block 200 to await another 

bar 152 is n^ck-base^"a"use^ ^e effect repre- 45 user action. However, if the program concludes in decision 

s ented the reb y , setup alfcTte block 202 that a clip was not dropped on a timeline, the logic 

^^^^E^^^^^^^^^^^^^Hj^^^J^m flow advances to a decision block 210. where the program 

track- effect ^flipp 150 independent of the clips that are determines whether or not a clip has been selected on the 

loaded on the track. These clips can then readily be changed timeline. If so. the logic flow advances to a decision block 

or modified, enabling the user to observe the effect applied 50 212. to determine whether the user is moving the selected 

to each clip or portion of a clip that is moved under effect bar clip on the timeline. If so, the logic advances to a block 214, 

152. where the program assigns the moved clip priority over 

In FIG. 8, track effect region 150 is modified to show existing active clips and globally redetermines the active- 
stacked track-based effects. In this Figure, an effect bar 154 ness for the timeline. After reevaluating the clip activeness, 
has been added above effect bar 152 and extends beyond the 55 the logic flow advances to end block 242. At this point, the 
right end of effect bar 152. The stacking order of effect bars program normally loops back to start block 200 and awaits 
152 and 154 controls the order in which they are applied to further user actions. 

modify any clips or portions thereof underlying both effect In decision block 212, the program can also conclude that 

bars. Specifically, the effect represented by effect bar 152 a selected clip was not moved, and the logic flow advances 

will be applied first, e.g., to the first part of clip 142, and men 60 to a decision block 216, where the program will ascertain 

the second effect represent by effect bar 154 will be applied. whether the user has slid the selected clip along the timeline. 

Since effect bar 154 extends beyond effect bar 152, only the If so. the logic advances to a block 218, to redetermine the 

effect it represents will apply to the portion of any clip activeness of the sliding clip in context with existing clips 

underlying the portion of effect bar 154 that extends beyond and globally reevaluate the activeness of the timeline. Once 

the end of effect bar 152. This arrangement provides con- 65 the evaluation is complete, the logic advances to end block 

siderable versatile control over the portions of any clip to 242, and then loops back to start block 200 to await yet 

which an effect applies. another user action. However, the program can also con- 
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elude that a selected dip was not slid on the timeline in 
decision block 216; in response, the logic advances to a 
decision block 220 as shown in FIG. 12. 

Referring now to FIG. 12 and decision block 220. the 
program determines whether the user is rolling the active- 
ness of the selected cup left or right within the constraints of 
the clip's existing activeness edit points. If so. the logic 
advances to a block 221 where the program redetermines 
the activeness of the selected clip by giving it priority over 
existing clips on the timeline. Next, the logic advances to a 
block 224. where the program globally redetermines the 
activeness of the timeline and binds the selected clip's 
activeness relative to the edit points. After completing these 
determinations, the logic advances to end block 242. looping 
back to start block 200 to await another user action. 

In decision block 220. the program may conclude that the 
clip was not rolled. If not. the logic advances to a decision 
block 226, where the program determines whether the user 
has selected the clip for trimming. If so, the logic advances 
to a block 228, and the program trims the selected cup. After 
trimining. the activeness cf the timeline is globally redeter- 
mined using the selected clip and all existing clips on the 
timeline. Once this evaluation is performed, the logic 
advances to end block 242 and loops back to start block 200 
to await further user actions. 

Each time that the activeness of the timeline is 
redetermined, the result depends upon the nature of the 
change effected by the user (whether trimming, inserting, or 
deleting a clip) and the status of one or more related 
parameters mat have been selected by the user. For example, 
if a clip is dropped onto the timeline over an existing clip, 
the redetermination of activeness will either move the exist- 
ing clip to make room for the new clip if a **ripple mode" 
parameter is selected, or will delete the existing clip if not 
However, since the rules applied for redetermining the 
activeness of the timeline are outside the scope of the present 
invention, there seems little reason to discuss such details in 
this disclosure. 

If the program concludes that a user is not trimming a 
selected clip in decision block 226. the logic advances to a 
decision block 230, where the program determines whether 
the user has activated the selected clip. If so, the logic 
advances to a block 232. where the program activates the 
selected clip and globally redetermines the activeness of the 
clip and the timeline. After the evaluation of activeness has 
been completed, the logic advances to end block 242 and 
back to the start of the procedure. 

In decision block 230. the program may conclude that the 
user was not activating the selected clip, and if so. advances 
to a decision block 234. Referring now to FIG. 13 and 
decision block 234. the program determines whether the 
user has deactivated the selected clip. If so. the logic 
advances to a block 236, where the program deactivates the 
selected clip and globally redetermines the activeness of the 
timeline with regard to any remaining active clips. Next, the 
logic advances to end block 242 and loops back to start 
block 200 to await another action by the user. 

However, if in decision block 234 the program determines 
that a user is not deactivating a selected dip. the logic 
advances to a decision block 238. to determine whether a 
user is 'filling the selected dip with activeness." If so. the 
logic advances to a block 240 and the program makes active 
(fills in) those parts of the clip where mere is no current 
activeness along the timeline from other existing cups. Once 
the clip activeness is filled in. the logic advances to end 
block 242. where the program flow loops back to start block 
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200 and awaits a further action by a user. However, if the 
program concluded in decision block 238 that a user has not 
selected a clip in which the activeness if to be filled, the logic 
advances to end block 242. where the program loops back to 

3 start block 200 to await another action by the user. 

Referring back to FIG. 12 and decision block 210. if the 
program concludes that the user had not selected a clip on 
the timeline, then the logic advances to a decision block 244. 
Referring now to FIG. 14 and decision block 244. the 

10 program determines if an edit point has been selected on the 
timeline. If so. the logic advances to a decision block 246. 
where the program determines if the user has moved the 
selected edit point. Once the program concludes an edit 
point was selected for moving, the logic advances to a block 

15 248. which provides for moving the selected edit point to the 
new location and then globally redetermines the activeness 
for the timeline. Next the logic advances to end block 242. 
with the subsequent loop back to start block 200. 
In dedsion block 246. the program may determine that the 

20 user is not moving the selected edit point, and if so. the logic 
advances to a dedsion block 252. In this decision block, the 
logic determines whether the user has added a transition to 
the selected edit point If no transition is added, the logic 
advances to end block 242 and loops back to start block 200. 

25 Alternatively, in decision block 252, the program may 
dedde that the user has added an edit point transition, and 
if so. the logic advances to a function block 254. where the 
program adds color coded activeness bars representing the 
transition to the clips immediately adjacent to the selected 

30 edit point. Once the activeness bars have been applied, the 
logic advances to end block 242. looping back to start block 
200 to await a further action by the user. 

Referring to decision block 252. if an edit point was not 
selected on the timeline, the logic advances to a decision 
block 256. Referring now to dedsion block 256 in FIG. 15. 
the program determines whether a timeline region was 
selected by the user. If so, the logic advances to a decision 
block 258, to determine if the user is activating the selected 

^ region. Once the program concludes the region has been 
activated, the logic advances to a block 260. In block 260. 
the program globally redetermines the selected region's 
timeline. * After performing this evaluation, the logic 
advances to end block 242. 

45 At decision block 258. the program may determine that 
the user has not activated the sdected region and will then 
advance to a decision block 264. In decision block 264, the 
program determines whether the user has deactivated the 
selected region. If so, the logic advances to a block 266. to 

so deactivate the sdected region. After deactivation, the logic 
advances to end block 242. However, a negative response to 
dedsion block 264 indicates that the user did not deactivate 
the selected region. In mis case, the logic advances to a 
dedsion block 268. where the program determines whether 

35 the user has selected filling in the activeness of the region. 
If so, the logic advances to a block 270. in which the region 
is made active in those areas where there is no conflict with 
current active regions on the timeline. 

After filling in the selected region, the logic advances to 

6o end block 242. Alternatively, if the program concludes in 
dedsion block 268 that the user is not filling the sdected 
region with activeness. then the logic advances to end block 
242. where the program flow loops back to start block 200 
to await further user actions. 

63 Referring to dedsion block 256. if the program deddes 
that a region has not been sdected. the logic advances to a 
dedsion block 272. In this dedsion block, the program 
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determines if an active bar has been selected. Referring to 
FIG. 16 and decision block 272. if the program concludes 
that an active bar has not been selected, then the logic 
advances to end block 242. Alternatively, if (he program 
concludes that an active bar has been selected, the logic 5 
advances to a decision block 274, where the program 
determines if the user has moved the selected active bar. If 
so. the logic advances to a block 276. In block 276. program 
moves the active bar to the new location selected by the user 
and globally redetermines the activeness of the timeline. 1Q 
After the activeness is redetermined, the logic advances to 
end block 242. looping back to start block 200 to await still 
further user actions. 

A negative response to decision block 274 indicates that 
the user is not moving the selected active bar. In response, 15 
the logic advances to a decision block 278. where the 
program determines if the user is trimming the bar. If so. the 
logic advances to a block 280 to trim the selected active bar 
and then globally redetermines the activeness of the time- 
line. Afterwards, the logic advances to end block 242 and 20 
back to start block 200. 

In decision block 278. if the user is not trimming the 
selected active bar. the logic advances to a decision block 
282. where the program determines whether the user wants 
to delete the selected active bar. If the bar is not selected for 25 
deletion, the logic advances to end block 242. Alternatively, 
if the program concludes mat the user does want to delete the 
selected active bar. the logic advances to a decision block 
284. This decision block determines whether the selected 
active bar is a transition bar, and if so, the logic advances to 30 
a block 286. In block 286. the program converts the transi- 
tion bar into an edit point by trimming the clips. After 
creating the edit point, the logic advances to end block 242 
However, if the program concludes in decision block 284 
that the selected active bar is not a transition bar, the logic 35 
advances to a block 288. In block 288. the program deletes 
the selected active bar. Once the bar is deleted, the logic 
advances to end block 242, looping back to start block 200 
to await yet another action by the user. 

Turning now to FIGS. 17 and 18. a clip and track based 40 
effects flow chart is illustrated. In FIG. 17, the logic begins 
with a start block 300 and advances to a decision block 302. 
to determines whether a section on the timeline has been 
selected, i.e., has a clip, a time span, or any other object on 
the timeline been selected that defines a region or time 43 
duration on the timeline. If no section on the timeline has 
been selected, the logic advances to a block 304, in which 
the program disables the clip and track effect buttons (or any 
other applicable user interface mechanism for selecting 
effects). After disabling the buttons, the logic loops back to 50 
decision block 302 to await the selection of something on a 
timeline by the user. Once a selection has been made on the 
timeline, the logic advances to a block 306. where the 
program enables the clip and the track effect buttons. After 
the buttons are enabled, the logic advances to a decision 55 
block 308. In this decision block, the program determines if 
a clip has been selected. If the response is affirmative, the 
logic advances to a decision block 310, in which the 
program decides whether the clip editing feature has been 
selected. If it has. the logic advances to a block 312, to edit 60 
the clip and clip effect After editing, the logic advances to 
a decision block 326 as shown in FIG. 18. Further, if the 
program concluded in decision block 310 that a clip was not 
selected for editing, then the logic advances to decision 
block 326. w 

In the response to decision block 308 is negative, the logic 
advances to a decision block 314. where the program 



determines whether the user has selected a clip filter. If so, 
the logic advances to a decision block 316. In decision block 
316, the program determines whether clip filter editing has 
been selected. If this type of editing is selected, the logic 
advances to a block 318 to edit the clip filter effect. Once the 
clip effect is edited, the logic advances to decision block 326 
in FIG. 18. Further, if the program concludes in decision 
block 316 that a clip filter was not selected for editing, the 
logic also advances to decision block 326. 

In decision block 314. the program may conclude that a 
clip filter was not selected and then advances to a decision 
block 320. At this point, the program determines whether a 
track filter was selected by the user. If the filter was selected, 
the logic advances to a decision block 322. to determine 
whether the track filter was selected by the user for editing. 
If so. the logic advances to a function block 324. and the 
program edits the track filter effect accordingly. Once the 
track filter effect is edited, the logic advances to decision 
block 326 in FIG. 18. Further, if the program concludes a 
track filter was not selected in decision block 320. the logic 
also advances to decision block 326. 

At decision block 326 in FIG. 18. the program determines 
if a clip button has been pressed by a user. If so, the logic 
advances to a block 328. where the program applies the 
selected clip effect. After the effect has been applied, the 
logic advances to an end block 334 and loops back to start 
block 300 to await another action by the user. In decision 
block 326. the program may conclude that a clip button was 
not pressed and. in response, the logic advances to a decision 
block 330. In decision block 330. the program determines 
whether a track button has been pressed by the user. If so, the 
logic advances to a block 332, where the program applies the 
selected track effect Once the track effect has been applied, 
the logic advances to end block 334. where the program 
loops back to start block 300 to await yet another action by 
the user. Further, the same result occurs if the program 
concludes in decision block 330 that a track button was not 
pressed. 

Although the present invention has been described in 
connection with the preferred form of practicing it those of 
ordinary skill in the art will understand that many modifi- 
cations can be made thereto within the scope of the claims 
that follow. Accordingly, it is not intended that the scope of 
the invention in any way be limited by the above description, 
but instead be determined entirely by reference to the claims 
that follow. 

The invention in which an exclusive right is claimed is 
defined by the following: 

1. A method for indicating activeness of a clip to a user 
who is composing and/or editing a multimedia work that 
includes a plurality of dips, comprising the steps of: 

(a) displaying a timeline for at least a portion of the 
multimedia work, said timeline graphically represent- 
ing the portion of the multimedia work and graphically 
illustrating a plurality of tracks for a time interval 
corresponding to that represented by the timeline; 

(b) enabling the user to select a clip as active, indicating 
that said clip is to contribute to an output comprising 
the multimedia work; and 

(c) marking the clip selected by the user as active with a 
visual indicator, so that by inspection of the visual 
indicator and a position of the clip on one of the 
plurality of tracks, the user can determine a time during 
which the clip contributes to a content of the multime- 
dia work. 

2. The method of claim 1, further comprising the steps of 
enabling the user to select a portion of the clip as active; and. 
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marking the portion of the clip selected as active with the 10. The system of claim 9, wherein each clip is repre- 

visual indicator, the position of the visual indicator relative sented by a rectangle disposed on one of the timelines, a 

to the clip indicating the portion of the clip that is active and duration of said clip being indicated by a relative size of at 

contributes to the multimedia work. least one dimension of the rectangle on the monitor, said 

3. The method of claim 2, wherein each clip is represented 5 visual indicator comprising a colored bar disposed adjacent 
by a rectangle disposed on one of the timelines, a duration onc 0 f m e rectangle, a length of the colored bar and 
of said clip being indicated by the a relative size of the position of the colored bar relative to the rectangle indicat- 
rectangle in at least one dimension, said visual indicator ing mc duration of the portion of the clip that is active, 
comprising a colored bar disposed adjacent one edge of the u system Q f daim o, wherein the machines ins true- 
rectangle, a length of the colored bar and position of the 1Q aons futtha CflUSC me processor to automatically make 
colored bar relative to the rectangle indicating the duration actiye portion Qf a ^ added {Q one of mc ^ 

of the portion of the clip that inactive. . does not overlap any portion of another clip that is already 

4. The method of claim 2. further comprising the step of ^ 

automatically making active any portionof a clip added I to > machines instruc- 

one of the timelines, where said portion does not overlap any . _ , J . 

portion of another clip that is already active. 15 uons me P"** 55 " to: 

5. The method of claim 1. further comprising the steps of (a) enable the user to select two cups to be simultaneously 
enabling the user to select two clips to be simultaneously active at one time when one of a transition and an 
active at one time when one of a transition and an overlay overlay is to occur between the two clips; and 

is to occur between the two clips; and, providing a visual ^ provide a visual indicator of the duration of said one 

indicator of the duration of said one of the transition and 20 of &e transition **& overlay between the two dips, 

overlay between the two clips. 13 S y StC m of claim 12. wherein the visual indicator 

6. The method of claim 5. wherein the visual indicator comprises a ban 

comprises a color bar having a color that indicates a specific £ The ^ of claim 12 , wherein the color bar has a 

typeofsaidoneofmetransitionandmeoverlayrxtw^ ^ ^ a spcdfic ^ of saW m of me 

K iiBe to facilitate determining whether to make a portion of a multimedia work when composing and/or 

cup active or inactive. editing the multimedia work, said method comprising the 

8. A system for indicating activeness of a clip to a user ^ steps of: 

composing and/or editing a multimedia work that includes a (a) displaying a timeline for at least a portion of the 

plurality of clips, comprising: multimedia work, said timeline graphically represent- 

(a) a monitor for displaying the multimedia work and for ing the portion of the multimedia work and graphically 
displaying screens used to compose and edit the raul- illustrating at least one track for a time interval corre- 
timedia work; 35 spending to that represented by the timeline; 

(b) a memory for storing a plurality of machine instruc- (b) enabling the user to select an effect that will apply to 
tions defining a software program for composing and the multimedia work and to select a time interval over 
editing the multimedia work; and which the effect will be applied, said effect being 

(c) a processor, coupled to the memory and to the monitor, associated with a selected track of the timeline by the 
for executing the machine instructions stored in the 40 user and being applied to only a selected portion of the 
memory, said machine instructions causing the proces- track; and 

sor to implement the following functions; (c) providing an effect indicator that indicates the portion 

(i) display a timeline for at least a portion of the 0 f the track to which the effect applies and the time 
multimedia work on the monitor, said timeline interval of the effect. 

graphically representing the portion of the multime- 45 itf. The method of claim 15. wherein the track includes an 

dia work and graphically illustrating a plurality of effect region that extends adjacent to the track, said effect 

tracks for a time interval corresponding to that indicator being disposed within the track region, 

represented by the timeline; 17. The method of claim 15, wherein the effect indicator 

(ii) enable the user to make a clip as active, indicating comprises a colored bar, a length of the colored bar and a 
that said clip does not contribute to an output com* 50 position of the color bar relative to the track indicating the 
prising the multimedia work, or inactive, indicating time interval over which the effect is applied. 

that said clip does not contribute to said output; and 18. The method of claim 17, wherein the effect is applied 

(iii) mark the dip selected by the user as active with a to a portion of any clip positioned on the track with which 
visual indicator shown on the monitor, so that by the effect is associated, adjacent the colored bar. but is 
inspection of the visual indicator and a position of 55 otherwise independent of any clip. 

the clip on one of the plurality of tracks that are jo. The method of claim 17. further comprising the step 

visible on the monitor, the user can determine a time 0 f enabling the user to modify the time interval over which 

during which the clip contributes to the output com- the effect is applied, including modifying a point in time at 

prising the multimedia work. which the effect is applied and its duration. 

9. The system of claim 8. wherein the machines instruc- 50 20, The method of claim 15, further comprising the step 
tions further cause the processor to: enabling the user to associate a plurality of effects with the 

(a) enable the user to select a portion of the clip as active; track, each effect having a user selectable time interval that 
and is independent of the time interval of other effects associated 

(b) mark the portion of the clip selected as active with the with the track. 

visual indicator, the position of the visual indicator 65 21. Hie method of claim 20, wherein the plurality of 

relative to the cup indicating the portion of the clip that effects are represented by corresponding color bars that are 

is active. disposed adjacent to the track with which the plurality of 



04/03/2004, EAST version: 1.4.1 



5/781 

17 

effects are associated, a length and a position of each of the 
color bars indicating the time and duration at which each is 
applied to the track. 

22. The method of daim 21, wherein predefined different 
colors are used for the color bars to represent different 5 
classes of effects, each predefined color representing a 
different class of effect 

23. The method of claim 21 . wherein an order in which the 
plurality of effects are applied determines a result in the 
multimedia work, said color bars representing the plurality , 0 
of effects being positioned relative to each other so as to 
indicate the order in which the effects are applied. 

24. A system for indicating effects applied to a specific 
portion of a multimedia work when composing and/or 
editing the multimedia work, comprising: l5 

(a) a monitor for displaying the multimedia work and for 
displaying screens used to compose and edit the mul- 
timedia work; 

(b) a memory for storing a plurality of machine instruc- 
tions defining a software program for composing and 20 
editing the multimedia work; and 

(c) a processor, coupled to the memory and to the monitor, 
for executing the machine instructions stored in the 
memory, said machine instructions causing the proces- 
sor to implement the following functions: 25 

(i) display a timeline for at least a portion of the 
multimedia work, said timeline graphically repre- 
senting the portion of the multimedia work and 
graphically illustrating at least one track for a time 
interval corresponding to that represented by the 30 
timeline; 

(ii) enable the user to select an effect that will apply to 
the multimedia work and to select a time interval 
over which the effect will be applied, said effect 
being associated with a selected track of the timeline 35 
by the user and being applied to only a selected 
portion of the track; and 

(iii) provide an effect indicator that indicates the por- 
tion of the track to which the effect applies and the 
time interval of the effect 40 

25. The system of claim 24, wherein the track with which 
the effect is associated includes an effect region that extends 
adjacent to the track, said effect indicator being disposed 
within the track region. 

26. The system of claim 24, wherein the effect indicator 45 
comprises a colored bar. a length of the colored bar and a 
position of the color bar relative to the track indicating the 
time interval over which the effect is applied. 

27. The system of claim 24. wherein the effect is applied 

to a portion of any clip positioned on the track with which 50 
the effect is associated, adjacent the colored bar. but is 
otherwise independent of the clip. 

28. The system of claim 24, wherein the machine instruc- 
tions further cause the processor to enable the user to modify 
the time interval over which the effect is applied, including 55 
enabling the user to modify a point in time at which the 
effect is applied and its duration. 

29. The system of claim 24. wherein the machine instruc- 
tions further cause the processor to enable the user to 
associate a plurality of effects with the track, each effect 60 
having a user selectable time interval that is independent of 
the time interval of other effects associated with the track. 

30. The system of claim 29, wherein different colors are 
used for the color bars to represent different classes of 
effects. 65 

31. The system of claim 29, wherein an order in which the 
plurality of effects are applied determines a result in the 
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multimedia work, said color bars representing the plurality 
of effects being positioned relative to each other so as to 
indicate the order in which the effects are applied. 

32. A method for indicating a plurality of effects that are 
applied to clips comprising a multimedia work, when com- 
posing and/or editing the multimedia work, said method 
comprising the steps of: 

(a) displaying a timeline for at least a portion of the 
multimedia work, said timeline graphically represent- 
ing a portion of the multimedia work and graphically 
illustrating at least one track on which is disposed at 
least one clip; 

(b) enabling the user to select a plurality of different 
effects mat will apply to said clip of the multimedia 
work and to select a portion of the clip to which each 
effect will be applied; and 

(c) providing an effect indicator for each effect that 
indicates the portion of the dip to which the effect 
applies and the time interval during which the effect 
applies. 

33. The method of claim 32, wherein the effect indicator 
for each clip comprises a colored bar, a length of the bar and 
its relative position adjacent the clip indicating the portion of 
the clip to which the effect applies. 

34. The method of claim 33. wherein the colored bars 
representing each of the plurality of effects selected by the 
user for application to the clip are stacked adjacent to the 
clip. 

35. The method of claim 34, wherein an order in which the 
colored bars are stacked indicates a sequence in which the 
effects are applied to the clip to produce a result 

36. The method of claim 32. wherein the plurality of 
effects are associated with the clip to which the plurality of 
effects apply, so that moving the dip along the timeline 
causes movement of the effects applied to the clip. 

37. The method of claim 32, further comprising the step 
of displaying a descriptive label when a cursor is moved 
over any effect indicator, said descriptive label identifying a 
class of effect that is represented by the effect indicator. 

38. A system for indicating a plurality of effects that are 
applied to dips comprising a multimedia work, when com- 
posing and/or editing the multimedia work, comprising: 

(a) a monitor for displaying the multimedia work and for 
displaying screens used to compose and edit the mul- 
timedia work; 

(b) a memory for storing a plurality of machine instruc- 
tions defining a software program for composing and 
editing the multimedia work; and 

(c) a processor, coupled to the memory and to the monitor, 
for executing the machine instructions stored in the 
memory, said machine instructions causing the proces- 
sor to implement the following functions: 

(i) display a timeline on the monitor for at least a 
portion of the multimedia work, said timeline graphi- 
cally representing the portion of the multimedia 
work and graphically illustrating at least one track on 
which is represented at least one clip; 

(ii) enable the user to select a plurality of different 
effects that will apply to said clip of the multimedia 
work and to select a portion of the clip to which each 
effect will be applied; and 

(iii) provide an effect indicator to be displayed on the 
timeline that indicates the portion of the dip to 
which the effect applies and the time interval within 
that of the clip, during which the effect applies. 

39. The system of claim 38, wherein the effect indicator 
for each clip comprises a colored bar. a length of the bar and 
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its relative position adjacent the clip indicating the portion of 
the dip to which the effect applies. 

40. The system of claim 39. wherein the colored bars 
representing each of the plurality of effects selected by the 
user for application to the clip are stacked adjacent to the 
clip. 

41. The system of claim 39, wherein an order in which the 
colored bars are stacked indicates a sequence in which the 
effects are applied to the clip to produce a result. 

41 The system of claim 38, wherein the plurality of 
effects are associated with the clip to which the plurality of 
effects apply, so that movement of the clip along the timeline 
causes movement of the effects applied to the clip. 

43. The system of claim 38, wherein the machine instruc- 
tions further cause the process to display a descriptive label 
on the monitor when a cursor is moved over any effect 
indicator, said descriptive label identifying a class of effect 
that is represented by the effect indicator. 

44. An article of manufacture adapted to be used with a 
computer for indicating activeness of a clip to a user who is 
composing and/or editing a multimedia work mat includes a 
plurality of clips, comprising: 

(a) a nonvolatile memory device; and 

(b) a plurality of machine instructions stored on the 
nonvolatile memory device, said machine instructions, 
when executed by the computer, causing it to: 

(i) display a timeline for at least a portion of the 
multimedia work* said timeline graphically repre- 
senting the portion of the multimedia work and 
graphically illustrating a plurality of tracks for a time 
interval during the multimedia work corresponding 
to that represented by the timeline; 

(ii) enable the user to make a cup active, indicating that 
said clip is to contribute to an output comprising the 
multimedia work, or inactive, indicating that said 
clip does not contribute said output; and 

(iii) mark the clip selected by the user as active with a 
visual indicator, so that by inspection of the visual 
indicator and a position of the clip on one of the 
plurality of tracks, the user can determine a time 
during which the clip contributes to a content of the 
multimedia work. 

45. An article of manufacture adapted to be used with a 
computer for indicating effects applied to a specific portion 
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of a multimedia work when composing and/or editing the 
multimedia work, comprising: 

(a) a nonvolatile memory device; and 

(b) a plurality of machine instructions stored on the 
5 nonvolatile memory device, said machine instructions. 

when executed by the computer, causing it to: 

(i) display a timeline for at least a portion of the 
multimedia work, said timeline graphically repre- 
senting the portion of the multimedia work and 

to graphically illustrating at least one track for a time 

interval corresponding to that represented by the 
timeline; 

(ii) enable the user to select an effect that will apply to 
the multimedia work and to select a time interval 

15 over which the effect will be applied, said effect 

being associated with a selected track of the timeline 
by the user and being applied to only a selected 
portion of the track; and 

(iii) provide an effect indicator that indicates the por- 
20 tion of the track to which the effect applies and the 

time interval of the effect 
46. An article of manufacture adapted to be used with a 
computer for indicating a plurality of effects that are applied 
to clips comprising a multimedia work, when composing 
25 and/or editing the multimedia work, comprising: 

(a) a nonvolatile memory device; and 

(b) a plurality of machine instructions stored on the 
nonvolatile memory device, said machine instructions, 
when executed by the computer, causing it to: 

30 (i) display a timeline for at least a portion of the 
multimedia work, said timeline graphically repre- 
senting a portion of the multimedia work and graphi- 
cally illustrating at least one track on which is 
disposed at least one cup; 

35 (ii) enable the user to select a plurality of different 
effects that will apply to said clip of the multimedia 
work and to select a portion of the clip to which each 
effect will be applied; and 
(iii) provide an effect indicator for each effect that 

40 indicates the portion of the clip to which the effect 

applies and the time interval during which the effect 
applies. 
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[57] ABSTRACT 

Parts for a multimedia title may be extended indefinitely 
relative to time with a user interface tool A part for a 
multimedia title is provided with edges having a wrap 
handle affixed to one edge. By dragging and dropping the 
wrap handle to a wrap corral, the part will play or be visible 
indefinitely. 
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METHOD AND SYSTEM FOR A 
MULTIMEDIA APPLICATION 
DEVELOPMENT SEQUENCE EDITOR 
USING A WRAP CORRAL 

CROSS REFERENCE TO RELATED 
APPLICATIONS 

This Application is related in subject matter to the fol- 
lowing Applications filed concurrently herewith and 
assigned to a common Assignee: 

Application Ser. No. 08/625,085 filed by King, et al., 
entitled "Method and System for a Multimedia Application 
Development Sequence Editor"; application Ser. No. 
08/623,727 filed by King, et al., entitled "Method and 
System for a Multimedia Application Development 
Sequence Editor Using Spacer Tools"; application Ser. No. 
08/625,084 filed by King, et al., entitled "Method and 
System for a Multimedia Application Development 
Sequence Editor Using Synchronizing Tools"; and applica- 
tion Ser. No. 08/622,909 filed by King, et al., entitled 
"Method and System for a Multimedia Application Devel- 
opment Sequence Editor Using Time Event Specifiers". 

The foregoing co-pending Applications are incorporated 
by reference. 

TECHNICAL FIELD OF THE INVENTION 

The present invention relates in general to computer 
software, and in particular, to a method and system for the 
creation of multimedia titles with a user friendly sequence 
editor using a wrap corral. 

BACKGROUND OF THE INVENTION 

Multimedia is a term used in connection with computer 
systems to refer to software that processes, i.e., creates, 
edits, displays, synchronizes, etc., one or more files of 
time-based data. A multimedia computer system is designed 
to present various materials in various combinations of text, 
graphics, video, image, animation, audio, etc. Multimedia 
systems commonly comprise a combination of hardware and 
software including a personal computer, a CD-ROM drive, 
software editors, etc. The data for multimedia presentations 
are generally stored on hardware memory devices, such as 
a magnetic disc drive, a ROM compact disk player, video 
tape device, etc. 

Application developers utilizing such a multimedia sys- 
tem may create programs that allow end users to view a 
multimedia presentation and manipulate various controls. 

One of the most difficult problems that face creators/ 
developers of multimedia titles is synchronization of time 
and events. For example, a developer must be able to 
synchronize audio with a video presentation and provide 
images that appear and disappear over time. In many com- 
plex real world examples it is desirable for two or more 
multimedia items to be synchronized based on one or more 
of the following: an event; a relative time relationship (i.e., 
A Before B, A After B, A coincident with B); and an absolute 
time. 

One product attempting to deal with time synchronization 
is Director from Macromedia. Director specifies time on a 
frame based metaphor. A Score is used in Director to specify 
when things occur in a movie (or title). The Score is a 
sequence of frames that hold cells of an animation and the 
indication to start playing audio or video. Things are speci- 
fied linearly in Director, Frames are sequenced in order 
unless software code is written to manually go back to a 



19,519 

2 

previous frame. Frames show sequentially one after the 
other. To create an animation, for example, a developer must 
manually associate each cell in the animation with a frame 
in the Score. There is no convenient way to cause things to 

5 occur in the relative manner. If two animations are to 
co-occur, corresponding frames must be manually specified 
on an individual basis. A Time tool can be used to specify the 
playing of a video; however, nothing else can be processed 
(i.e., animation, other videos, etc.) while the video is play- 

30 ing. There is no point and click mechanism for wrap; code 
must be written to cause a wrap. 

Another product, Premier from Adobe, is a video editing 
product. It has a facility called the Construction Window 
(CW) where clips (i.e., video, scan images, QuickTime, 

15 movies, audio) are manipulated to create movies. The CW 
has channels, a timeline, and a time unit selector to adjust the 
scale of the ticks on the timeline. Pieces of video and images 
are put in channels and manually aligned (e.g. via dragging) 
with an absolute time and with other pieces of media. There 

20 is no notion of relative time or event time and no alignment 
helper co-tools. The notion of wrap does not exist for the 
creation of a movie. 

Finally, IconAuthor (IA) is a multimedia authoring tool 
which allows creation of applications that include time- 

25 based media, logic, database, etc. IA uses a flowchart 
metaphor to specify an application. Icons representing the 
playing of media, database access, comparisons, 
assignment, etc. are taken from a palette and connected into 
a flowchart. Processing in the application then follows the 

30 hierarchical structure defined in the flowchart. There is no 
editor to synchronize processing visually. Synchronization is 
done by checking logic and by corresponding logic that 
responds to events. 

35 Thus, there is a need for a method that simply and visually 
provides synchronization of multimedia parts in relation to 
event time, relative time, and absolute time and uses a wrap 
corral. 

SUMMARY OF THE INVENTION 

40 

The present invention disclosed herein comprises a 
method and apparatus for extending parts of a multimedia 
title indefinitely relative to time which substantially elimi- 
nates problems associated with prior time extending pro- 

45 grams. The present invention allows a user a simple and easy 
to use method for continuously playing or showing a part. 

In accordance with one aspect of the invention, a user 
interface tool is provided for extending parts of a multimedia 
title indefinitely relative to time. A graphical representation 

50 of the part is provided with edges and a dragging handle 
positioned on at least one of the edges. A user may then 
simply grab the dragging handle and drag the edge into a 
wrap corral. Once the edge has been placed into the wrap 
corral, the part will play or be visible indefinitely. 

55 It is a technical advantage of the present invention that a 
developer is no longer required to write code to replay or 
continuously show a multimedia part for a multimedia title. 

BRIEF DESCRIPTION OF TOE DRAWINGS 

60 

For a more complete understanding of the present inven- 
tion and for further advantages thereof, reference is now 
made to the following Detailed Description, taken in con- 
junction with the accompanying Drawings, in which: 
65 FIG. 1 is an illustration of a general computer architecture 
of a computer system within which the present invention 
may operate; 
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FIG. 2 is a graphical illustration of a sequence editor in can be thought of as templates for object types. Instances (or 

accordance with the present invention; objects) are particular run-time instantiations of a class. An 

FIG. 3 is an illustration of placement of a part within a object provides storage for variables defined in its class, 

channel; Each instance has its own version of those variables. When 

FIG. 4 is an illustration of the play tool; 5 a message is sent to an object, the object looks to methods 

FIG. 5 is an illustration of placement of an audio player defined J n lls class t0 determine how the ^ essa S e * "> be 

executed. 

part; 

mr- a v „„ u, c j,. f ♦ i • „ Classes are defined in terms of a class hierarchy. A class 

FIG. 6 is an illustration of the play tool in conjunction . . . LJ r , 

tko „, r ». inherits all vanables and methods from classes higher in the 

with the audio player part, mi.* l c j . i ^ • L • • * 

n „ „ . .„ : P , ,. , 10 hierarchy, referred to as super classes. The inheriting class 

FIG. 7 is an illustration of the placement of a video player may be {Q as & Qf chiJd of ^ superc]ass 

part ' The variables and methods inherited from a superclass are 

FIG. 8 is an illustration of placement of the play tool in available to the inheriting class as if they were defined 

conjunction with the video player part; within the inheriting class. Note that if a subclass has 

FIG. 9 is an illustration of placement of an image part; is multiple superclasses, all the variables and methods from 

FIG. 10 is an illustration of the use of a co-occur tool; each superclass are inherited. This is often referred to as 

FIG. 11 is an illustration of the use of a meet tool; multiple inheritance. 

FIG. 12 is an illustration of use of a hide spacer; There are two types of variables that may be defined 

FIG. 13 is an illustration of placement of a time event within a class - ^ e first are class variables, whose storage is 

specifier; 20 snare d by all instances of the defining class. Class variables 

FIG. 14 is an illustration of use of the move spacer; are tyP^Uy used for coordination between classes and 

rip k ;c o« .-iwtrotiVtn ~f «.<.» th* r *t* c^.^r- instances. The second are instance variables, which are 

rid. 15 is an illustration or use oi the rate spacer; ... ._ . . . _ ' . 

™ 0 1jC . , -ii . e u- j * variables specific to each particular instance. That is separate 

FIG. 16 is a schematic illustration of a multimedia title; , • n . J\ u** ■ . , c ^ 

, storage space is allocated to each instance variable of each 

and 25 ■ . 

instance 

FIG. 17 is a schematic illustration of a multimedia title w ' iL . , „ . 

, r t c Methods specify the action to be performed when a 

and use of an event specifier. . . . . • 4 t . r 

r message is sent to an object. When the message is sent to an 

DETAILED DESCRIPTION OF THE object, the message name is matched against method names 

INVENTION defined for that object. The method actually executed is the 

Referring first to FIG. 1, there is shown a general archi- method with the same name as the message that is defined 
tecture 18 of a system of the type within which the present lowest in that object's class hierarchy, 
invention operates. Architecture 18 comprises a main bus Messages can have parameters. When a message is sent, 
12, which serves to interconnect the various components, all actual parameters are fully evaluated before a method is 
including at least some of the following (and possibly other 35 located for that message. Methods have formal parameters, 
additional components): CPU 14, floating point unit 16, bus corresponding to the actual parameters in the message, and 
controller 28, video RAM 22, dynamic RAM 24, static temporary variables. Temporary variables may be created 
RAM 26, digital signal processor 29, internal hard disk 30, when the message is executed, and are available only with 
external memory device 32 (connected for example via a the scope of that method while the method is executing. 
SCSI port 34), external network devices 36 (communicating 40 Programs perform by sending messages to objects which 
for example over an ethernet network 38, and connected via invoke a method in an object's class or class hierarchy. 
SCSI port 34), a display device 40 (such as a CRT), printer Often, methods send messages to other objects, invoking 
42, keyboard 46, and pointing device 48 (such as a mouse, other methods. Typically, each method returns a result to the 
trackball, etc.). The interconnection and functioning of each sender. At the heart of many languages, and especially an 
of these components in an operating computer system are 45 object oriented language, is a program segment for inter- 
well known in the art. However, the function and signifi- preting and evaluating messages and keeping track of where 
cance of particular components will be discussed in further to return when methods call other methods, 
detail where appropriate. Object oriented programming environments can be imple- 

While the discussion that follows describes the invention . mented with many programming languages. Some common 

in terms of the software programming environment com- 50 examples of object oriented environments are: the C++ 

monly known as object oriented software design (such as a environment, the Smalltalk environment, and the ScriptX 

ScriptX from Apple Computer), it will be appreciated that environment. 

the functionality of the present invention is not limited to Software program code which employs the present inven- 

such environments. However, as support for one possible tion is typically stored in memory, such as the internal hard 

environment in which the program invention operates, a 55 disk 30, from which a developer may access the code. For 

description of certain relevant details of object oriented distribution purposes, the software program code may be 

programming environments is presented. embodied on any of a variety of known media for use with 

In an object oriented programming language, the funda- a data processing system, such as a diskette or CD-ROM, or 

mental data structures of the language are objects, program- may be distributed over a network of some type to other 

ming elements that contain a specification of both data and 60 computer systems for use by users of such other systems, 

actions. Data is specified in local variables, referred to as Such techniques and methods for embodying software code 

instance variables, contained within the object. Actions, or on media and/or distributing software code are well known 

methods, are sub-programs that operate on the variables. and will not be further discussed herein. 

Methods arc invoked by messages sent from one object to The present invention will now be discussed in conjunc- 

itself or other objects. 65 tion with an example and with reference to FIGS. 2-17. As 

There are two fundamental object constructs: classes and previously stated, one of the most difficult problems facing 

instances. Classes have a set of variables and methods and • creators of applications that incorporate multimedia is the 
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time and event synchronization problem. An example of this co-begin tool 66, a co-end tool 68, a co-occur tool 70, and 

might be where an application requires the following types a meet tool 72. 

of processing: 1) the pressing of a push button causing an The co-begin tool 66 allows parts which are selected, as 

animation to play; 2) during the course of the animation, a will be discussed later, to begin at exactly the same time, 

piece of audio is also played; 3) at the simultaneous ending 5 Similarly, the co-end tool 68 allows parts to end at exactly 

of the animation and audio, a video plays, the second half of the same time. The co-occur tool 70 allows parts to begin 

which plays at twice the rate of the first half; 4) independent and end at exactly the same times, while the meet tool 72 

of 1-3 above, an image appears ten seconds after the push allows one part to begin immediately upon the ending of 

button is pressed, remains visible for 20 seconds while another. The use of these tools will be discussed in greater 

moving across the screen, and then disappears; and 5) ]Q detail below. Beneath the toolbar 64 is a part starter 74 and 

shortly after the video begins an exit push button appears on an event starter 76. By clicking on the scroll arrows 78 and 

the display. 80, respectively, drop down boxes appear for the selection of 

Referring to FIG. 16, a real world example that might an appropriate, arbitrary part and an appropriate, arbitrary 

require this type of processing could be a computer based event. 

training application (a multimedia title 300) for a student in 1S Below the part and event starters 74 and 76, there is a 

an automotive service center. The student would click on an zoomer 82 and a time -line 84. The zoomer 82 allows the 

arbitrary part, a push button 302, on a computer to view setting of a time value for each tick interval on the time-line 

information about how to assemble a previously selected 84 - B Y moving a slider 86, a tick interval setting may be 

automobile part. The arbitrary event, i.e., clicking the push selected. As shown in FIG. 2, the tick interval setting is for 

button, would trigger a sequence of time synchronized 2Q 0 5 seconds. Therefore, each lick interval on time-line 84 

processing 304 starting with an animation part sequence 306 represents 0.5 seconds. 

that showed the sub-components of the automobile parts Directly below the zoomer 82 are a plurality of adjustable, 

connecting themselves together so the student could learn drag and drop tools for use by the developer. A play spacer 

how the componentry is assembled. The audio player (audio 88 allows the developer to indicate how long a time based 

part 308) that runs coincident to the animation would be an 25 media part (i.e., processing) is to play. A hide spacer 90 is 

oral description of how the assembling process is accom- used to hide a part that would otherwise be visible. A move 

plished. When the animation and audio complete, a video spacer 92 causes an object to move along a path while 

(video part 310) is played that gives a 360 degree view of the continuing to play or be visible. A rate spacer 96 changes the 

assembled auto part for perspective purposes and complete- rate that the media is played for the length of time of the 

ness of the student's background. Due to time constraints, 30 spacer. 

the second half of the video plays at twice the rate of the first Time event specifiers are also located beneath the zoomer 

half. Independent of the above processing, an image (image 82. A time event specifier 98 specifies an event to be 

part 312) is displayed 10 seconds into the playing of the generated at a particular point in time, as will be subse- 

animalion, and moves across the screen for 20 seconds. The quently discussed in greater detail. A periodic time event 

image is a welcome message wishing the student good luck. 35 specifier 94 specifies an event to be generated at a particular 

Also independent of the above processing, shortly after the point in time and at subsequent intervals thereto, 

video begins playing, an exit push button (push button part A bullpen 100 is used as a staging area for parts on a scene 

314) appears at the bottom of the screen allowing the student that have been selected previously. As shown in sequence 

to exit the video if it is deemed unnecessary to further editor screen 60, the parts for scene one are stored in the 

understanding, 40 bullpen 100. 

To solve the synchronization problem as described above, Below the time-line 84, are a plurality of visibility chan- 

a sequence editor in accordance with the present invention nels 102, 104, 106, 108 and 110. Corresponding to each 

is provided. The sequence editor is a mechanism that allows visibility channel 102 through 110, are thumbnail areas 112, 

a user to synchronize relative time, absolute time and event 114, 116, 118 and 120. The thumbnail areas are used to place 

time together in an easy to use manner. The solution assumes 45 parts which will be synchronized by the sequence editor. The 

that items for synchronization (such as an animation, video, visibility channels are used to specify the period of time that 

audio, image) have been previously identified and selected the part within the appropriate thumbnail area is visible or is 

by a user. There are several tools which allow selection of played. 

these types of items for synchronization and further descrip- As indicated above, the remaining FIGS. 3-17 will be 

tion is not required. 50 used to describe a real world example of a multimedia 

The sequence editor approach to synchronization starts application that an application developer might create. The 
with events. Events are unpredictable and often occur ran- example is the pressing of a push button to cause an 
domly. They can be caused by user interaction or other animation to play (an animation showing sub-components of 
asynchronous activity. Examples of events could be a push an automobile part connecting itself together). During the 
button being clicked, a video playing to completion, an 55 course of the animation, a piece of audio is also played (an 
absolute time being reached, or an error condition occurring. . oral description of how the assembling process is 
An event is defined by the program creator, and is a accomplished, plus some music). At the simultaneous end- 
notification of some occurrence of some thing. The sequence ing of the animation and audio, a video plays (a 360 degree 
editor organizes its time based specification by event. In view of the assembled part for perspective purposes and 
other words, a user specifies an event that will initiate a eo completeness of the student's background). The second half 
sequence of time based processing. In the example that of the video is to play at twice the rate of the first half, 
follows, the event is the clicking (selection of) a push button, Independently, an image will appear 10 seconds after the 
and the time based processing is the animation and media push button is pressed, remain visible and move across the 
related playing. screen for 20 seconds and then disappear (a welcome 

Referring now to FIG. 2, a sequence editor screen 60 is 65 message). Shortly after the video begins, an exit button 

shown. Directly beneath a title bar 62, is a toolbar 64. On the appears on the display (allows the student to exit the video 

toolbar 64 are a plurality of convenience tools including a if deemed unnecessary). 
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Referring to FIG. 3, the developer has set the zoomer 82 
for a two second interval. Therefore, each tick mark on the 
time-line 84 will indicate two seconds with each larger tick 
mark 120 indicating ten seconds and each largest tick mark 
122 will indicate twenty seconds. The developer has also 
dragged the animation part 124 from the bullpen 100, and 
dropped it into thumbnail area 112. Upon placement of the 
animation part 124 in the thumbnail area 112, the visibility 
channel 102 (See FIG. 2) is changed to a show block 103 
which indicates the animation part 124 can appear along the 
entire time-line 84. 

Referring to FIG. 4, the play tool 88 has been dragged and 
dropped onto the show block 103 as indicated by a play icon 
130 and a play area 131. By dragging a size handle 132 to 
line up under a 100 second tick mark 134 on the time-line 
84, one can see that the animation player will play from the 
moment the push button is clicked until 100 seconds of 
absolute time have elapsed. 

Referring to FIG. 5, an audio player part 140 has been 
dragged from the bullpen 100 and dropped into the thumb- 
nail area 114. As discussed above with reference to anima- 
tion player 124, upon dropping the audio player 140 into the 
thumbnail area 114 a visibility block 105 is placed in the 
visibility channel 104 (FIG. 2). The visibility block 105 
extends throughout the length of the time-line 84. For the 
case of a non-visual time based media part (i.e., audio), 
visibility implies that the part can be played. 

Referring to FIG. 6, the play tool 88 is dragged and 
dropped onto the visibility block 105 as indicated by a play 
icon 142 and a play area 143. Since a piece of audio plays 
for a finite period of time, the audio player part 140 can be 
seen to have an approximate duration of about 30 seconds. 
As will be discussed in greater detail, it is intended in this 
example to make the audio coincident throughout with the 
animation. The present invention allows such synchroniza- 
tion with a few simple steps. 

Referring to FIG, 7, a video player part 150 has been 
dragged from the bullpen 100 and dropped onto the thumb- 
nail area 116. As before, the visibility channel 106 (FIG. 2) 
is changed to indicate a show block 107 throughout the 
time -line 84. The play tool 88 is dragged and dropped onto 
the show block 107 (FIG. 8), as indicated by a play icon 152 
and a play area 153. As with the audio portion of audio 
player part 140, the video player part has a finite length and 
therefore plays for a specified amount of time which is 
shown as the play area 153. 

Referring to FIG. 9, the developer has selected Image 1 
part 160 from the bullpen 100 and dropped it into thumbnail 
area 118. The visibility channel 108 is then changed to a 
show block 109 throughout the time-line 84. 

As was stated in the real world example above, the 
animation and audio were to start and stop at the same time. 
By utilizing the present invention, the developer may simply 
ensure that this occurs. Referring to FIG. 10, by selecting 
first the animation player play area 131 and then the audio 
player play area 143 and then the co-occur tool 70, it is 
ensured that the animation player part and the audio player 
part will start and stop simultaneously. Since the audio was 
only about 30 seconds in length, the audio will obviously 
have to be replayed to reach the 100 second length of the 
animation. 

Similarly, referring to FIG. U, the video is to begin as 
soon as the animation ends. Thus, the developer first selects 
the animation player play area 131, then the video player 
play area 153 and, finally, the meet tool 72. By so doing, the 
developer has now ensured simply and quickly that the 
video will start upon the conclusion of the animation. 
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The use of the co-tools as described includes an important 
embodiment of this invention. These tools allow automatic 
implementation of time alignment of three types: 1) absolute 
time, 2) relative time, and 3) event driven time. In terms of 

5 background, it may be helpful to elaborate on absolute time 
(aT), relative time (rT), and event drive time (eT). Absolute 
time (aT) is time that has a fixed starling point. It may 
originally be specified as offset from some relative time (rT) 
or some event driven point in time (eT), but aT is specified 

1Q overtly herein with respect to a tick mark on the time-line. 
Given a particular time sequence, aTis declared independent 
of other parts. Relative time (rT) is time that has a starting 
(or ending or both) point relative to some other part (or 
parts). Relative time (rT) is specified overtly with respect to 
the starting or ending of some other occurrence of something 

15 and its specific value is not known or declared. Event driven 
time (eT) is time that has a starting point that is defined by 
the occurrence of some event which may or may not have 
any relationship with time. An event could be a user inter- 
action with a push button being selected or it could be an 

20 error condition. There are an infinite number of event types. 
Subsequent to specification of a part that is synchronized 
with eT, it is impossible to determine the point at which that 
time will occur. It will occur when that event occurs. As used 
herein, "initiating appropriate part behavior 5 ' means to cause 

25 the part to be, for example, displayed or played as appro- 
priate to the type of time based media part. It is to be 
understood that appropriate part behavior may encompass 
any behavior that the developer desires. 

Bringing rT back to our example, the synchronization of 

3 0 the audio and animation via the co-occur tool is an imple- 
mentation of specifying both the starting and ending rTs of 
the audio with respect to the animation. The synchronization 
of the video and animation via the meet tool is an imple- 
mentation of specifying the starting rT of the video with 

35 respect to the ending time of the animation. 

Bringing aT back to our example, the synchronization of 
the disappearance of the image at the absolute time mark of 
30 seconds on the time-line is an implementation of speci- 
fying the starting aT of the image disappearance with respect 

40 to an absolute time on the time-line. The specification is 
achieved by overtly dragging the hide block start time to the 
30 second tick mark. 

Bringing eT back to our example, the initiation of the 
entire sequence of time based processing starts at the 

45 unknown time of selection of the clicked event of the push 
button part specified by the part and event starter drop down 
lists. 

Referring now to FIG. 12, it is still required to synchro- 
nize the Image 1 part 160. As required by the example, 

50 Image 1 part 160 is to appear ten seconds after the animation 
starts, remain visible for 20 seconds while moving across the 
screen, and then disappear. Therefore, the developer drags a 
hide spacer 90 and drops it onto the show block 109, as 
indicated by a hide icon 180 and a hide area 181. For 

55 convenience purposes only, the example shown has a hide 
area width of a predetermined value of ten seconds. 
Therefore, the hide icon 180 will cause the Image 1 part 160 
to be hidden for the required ten seconds after the animation 
starts. The developer then grabs another hide spacer 90 and 

60 drops it onto the show block 109 as indicated by a hide icon 
182. Then, by grabbing a size handle 184 and dragging to a 
wrap corral 186, it is ensured that the Image 1 part 160 will 
disappear after being visible for 20 seconds and not 
reappear, as shown by hide area 183. A wrap corral can be 

65 considered as infinity in the present invention. By using the 
wrap corral, a developer can "hook" the end time of a show 
or play block and thus have it continue to infinity. 
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In addition, continuing to use the above example, the 
image which is a welcome message to the student is to move 
in a straight line across the screen from left to right during 
the 20 seconds that it is visible. To accomplish this, the 
developer would select the move spacer 92 from the palette 
of tools and drop it on the visibility block 109 as indicated 
by icon 210 and area 211 (FIG. 14). The visibility block 109 
(FIG. 12) is now shown as area 211 from an absolute time 
of 10 seconds until an absolute time of 30 seconds. Of 
course, this sequence is relative to the event time of the 
clicked event. The beginning (left edge) of the move spacer 
area 211 is dragged to the absolute lime of 10 seconds. The 
ending time (right edge) of the move spacer area 211 (size 
handle) is dragged to the absolute time of 30 seconds. This 
move spacer area 211 now indicates that during the course 
of its time (i.e., aT 10 until aT 30) the image will not only 
be visible, but will move along some path. To those knowl- 
edgeable in the art of animation and computer graphics, 
there are many well known mechanisms to define such a 
path. Macromedia's Director, for example, describes in their 
guide at pages 113 and 114 their specification technique to 
allow a user of their tool to specify a path for motion of an 
image. There is no claim herein of novelty in motion 
specification techniques. As a result, we will assume that 
"double clicking" the move spacer icon, once it is placed on 
the visibility block of the image, allows the user to specify 
a motion path for the image that is moving across the screen. 
The uniqueness herein is that once a path is specified, the 
sequence editor of the present invention has* provided an 
easy to use mechanism that synchronizes the movement 
across the screen with the length of time the move spacer 
occupies. 

Referring to FIG. 13, the developer drags and drops a time 
event specifier 98 onto the video play area 153 as indicated 
by a specifier icon 190. By positioning the time event 
specifier icon 190 (an arbitrary unique event) properly 
thereon, the developer may then trigger another unique 
event at the desired time. This unique event could then cause 
another sequence to occur or, as in our real world example, 
it could cause some other processing to make a push button 
appear, which allows for termination of the video, if desired. 

Referring to FIG. 17, the event specifier of FIG. 13 is 
graphically illustrated in use. As previously described with 
respect to FIG. 16, the multimedia title 300 is created. It is 
desired by the developer to generate a new arbitrary 
sequence 320 or a new arbitrary part 322 by a unique 
arbitrary event 324 (i.e., at a particular time, encountering 
the event specifier). Thus, as per the description above, the 
event specifier allows generation of the unique event. 

An addition to the above example, it may be desired to 
have the second half of the video played at twice the normal 
speed. This could be necessary due to the time requirements 
levied on the curriculum. To accomplish this, a developer 
would select the rate spacer 96 (see FIG. 15) from the palette 
of tools and drop it on the video play area 153 as indicated 
by icon 220 and area 222. Using normal drag techniques and 
the size handle, the area 222 is adjusted to occupy the second 
half of the video play area 153. Although it is not part of the 
present invention, the developer could, for example, then 
double click on the rate spacer icon 220 and a dialog box 
would appear to query the developer for the rate required. 
The second half of the video would then play at the new rate. 

Thus, the present invention provides heretofore unavail- 
able simplicity of use for the multimedia title developer. The 
developer has all the tools necessary to create a synchro- 
nized multimedia title in an easy to use visual interface type 
application. Events and parts can be synchronized using 



absolute time, relative time and/or event time which allows 
the developer a great deal of flexibility. The sequence editor, 
as taught herein, provides for the developer a view of the 
multimedia title from the perspective of its time structure as 
5 opposed to previously available data flow or user interface 
perspectives. 

Although the present invention has been described with 
respect to a specific preferred embodiment thereof, various 
changes and modifications may be suggested to one skilled 
3 o in the art, and it is intended that the present invention 
encompass such changes and modifications as fall within the 
scope of the appended claims. 

We claim: 

1. A multimedia application development sequence editor 
!5 for enabling a user of said sequence editor to specify a 
continuous display of a part of a multimedia title, compris- 
ing: 

a graphical user interface which provides an at a glance 
relational visual layout of a plurality of multimedia 
20 parts in terms of a time structure thereof; 

at least one spacer area graphically represented on said 
. graphical user interface, wherein each of said spacer 
areas is visually associated with a selected one of said 
parts; 

25 means, operable by said user, for dragging a graphical 
representation of an edge of a selected one of said 
spacer areas; 

means, operable by said user, for dropping said graphical 
representation of an edge into a wrap corral, wherein 
30 said wrap corral is a special-function area graphically 
depicted on said visual layout; and 
means, responsive to said dragging and dropping, for 
causing said selected part to be appropriately displayed 
continuously while the multimedia title is active. 
35 2. The multimedia application development sequence 
editor of claim 1, wherein said means for dragging a 
graphical representation of an edge of a selected one of said 
spacer areas further comprises: 

means, operable by said user, for dragging a wrap handle 
affixed to said edge. 

3. A method for enabling a user of a multimedia appli- 
cation development sequence editor to specify a continuous 
display of a part of a multimedia title with said sequence 
editor, comprising the steps of: 

providing a graphical user interface with an at a glance 
relational visual layout of a plurality of multimedia 
parts in terms of a time structure thereof; 
graphically representing at least one spacer area on said 
graphical user interface, wherein each of said spacer 
areas is visually associated with a selected one of said 
parts; 

dragging a graphical representation of an edge of a 
selected one of said spacer areas; 
55 dropping said graphical representation of an edge into a 
wrap corral, wherein said wrap corral is a special- 
function area graphically depicted on said visual lay- 
out; and 

responsive to said dragging and dropping, causing said 
60 selected part to be appropriately displayed continu- 
ously while the multimedia title is active. 

4. The method of claim 3, wherein said step of dragging 
a graphical representation of an edge of a selected one of 
said spacer areas further comprises: 

65 dragging a wrap handle affixed to said edge. 

5. A wrap corral for enabling a user of a multimedia 
application development sequence editor to specify a con- 
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tinuous display of a part of a multimedia title, wherein said 
wrap corral is a special-function area graphically depicted 
on an at a glance relational visual layout of a plurality of 
multimedia parts, said layout presented on a graphical user 
interface in terms of a time structure of said parts, compris- 5 
ing: 

means, operable by said user, for dragging a graphical 
representation of an edge of a selected one of at least 
one spacer areas graphically represented on said 
graphical user interface, wherein each of said spacer 10 
areas is visually associated with a selected one of said 
parts; 
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means, operable by said user, for dropping said graphical 
representation of an edge into the wrap corral; and 

means, responsive to said dragging and dropping, for 
causing said selected part to be appropriately displayed 
continuously while the multimedia title is active. 

6. The wrap corral of claim 5, wherein said means for 
dragging a graphical representation of an edge of a selected 
one of said spacer areas further comprises: 

means, operable by said user, for dragging a wrap handle 
affixed to said edge. 
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[57] ABSTRACT 

A method and apparatus, in which video clips (and option- 
ally also still image and audio clips) are stored as digital data 
in a computer memory, selected clips are displayed in 
elongated tracks on a display screen, and editing operations 
are performed on the clips in response to manipulation of 
displayed cursors and icons to assemble and preview an 
edited video program. Preferably, the inventive apparatus is 
a computer system programmed to display video, still 
image, and audio clips, and special effect icons, in tracks 
along a displayed time line. The system assembles a video 
program from stored clips in response to arrangement of 
displayed clips and special effect icons in a desired sequence 
along the time line. The computer system is preferably 
programmed: to select a new in or out point for a clip by 
positioning a cursor at an edge of the displayed clip and 
dragging the edge relative to the time line, to select a special 
effect transition between displayed clips by positioning a 
transition icon in a special track in alignment with overlap- 
ping portions of the clips, to select special effect parameters 
by manipulating an icon in a special effects track, to filter 
selected video clips with a mosaic filter having user- 
selectable time-varying filter characteristics, and to control 
superimposition of an overlay clip with a main clip in 
response to manipulation of a level control icon displayed in 
alignment with the overlay clip. 
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METHOD AND APPARATUS FOR VIDEO segments. The system also assembles (from the unedited 

EDITING WITH VIDEO CLIP video segments) and displays a sequence of the pictorial 

REPRESENTATIONS DISPLAYED ALONG A labels, which represent scenes (and transitions between 

TIME LINE scenes) of the edited program. The video segments are 

5 stored on video tape, as is the assembled video program .-The 

FIELD OF THE INVENTION system has controls which enable a user to specify a variety 

^ . . . . j., j.. of transitions between consecutive video segments. 

T*e invention is a computer-implemented video -editing Qther ^ ^ ms have beeQ d whicfa 

method and apparatus in which stored video clips are indude meanfi for ^ UQedited videQ ^ (oQ yideo 

displayed, and an edited video program is assembled from t or Iaser yideo disks)> and compuler means for ^^i. 

selected ones of the clips. The apparatus of the invention is « the di , of unedhed vjde0 d - (or frames 

a computer system programmed to perform video editing of videQ di } and erati an edit list in nse t0 

operations in response to manipulation of displayed menus use r-entered commands. 

and 1COns * For example, U.S. Pat. 4,746,994 (issued May 24, 1988, 

BACKGROUND OF THE INVENTION is t0 Ettlinger) discloses a computer-based video editing sys- 
tem in which unedited clips are stored on video tape record - 

Throughout this specification, the term "video clip" will efS or video disk p[ayers A computer syslem enables the 

denote one or more consecutive frames of video data, the user t0 control the video tape recorders and generate an edit 

term "still image clip" will denote data representing one or ^st. The computer system displays a sequence of menus 

more identical frames of still image data (or text), and the which prompt the user t0 per f orm various editing operations 

term "audio clip" will denote audio soundtrack data corre- (such ^ displaying desired framcs of the uned i ted clips, 

sponding to one or more video clips. A video clip (or a still shuttling frame-by-frame through a stored unedited clip, 

image clip) can be a scene of an edited video program. adding edils l0 an edit list) and playing back the of 

Throughout this specification, the expression "display of c ii ps defined by the edit list). The user may select various 

a clip" and denotes display (at a single time) of a represen- 25 ones of the editing operations by actuating a light pen. 

tation of the data defining the clip, unless qualified to read F or another example, U.S. Pat. 4,754,342 (issued Jun. 28, 

"sequential display of a clip". Similarly, "displayed clip" 1988) t0 Duffy ) discloses a computer-based video editing 

denotes a simultaneously displayed representation of the system in which unedited clips are stored on video disk 

data defining the clip, unless qualified to read "sequentially playerS- Mu >t generating an edit list, the user may command 

displayed clip". The expressions "sequential display" of a 3Q lne system t0 splice dips in accordance with the edit list and 

program, "sequential display" of a clip, and variations on play back an edited show By a conlrol conS ole with 

these expressions, denote the non-simultaneous (time- control buttons and a control dial, the user may command 

sequential) display of a sequence of representations of data the sys tem to display individual frames of the stored clips, 

sets, where the data sets collectively define a program or or « roir one or more clips (or m edited 0 f clips) 

CU P- 35 in forward or reverse motion, at any of a variety of speeds. 

Throughout this specification, including in the claims, the SUMMARY OF THE INVENTION 

expressions "edited video program," "video program," and oUMMAKi w 1HL IJN VbN HUM 

"program" are used interchangeably to denote a sequence of The invention is a method and apparatus for video editing, \ 

video clips or still image clips (or video clips and still image in which video clips (and optionally also still image clips J 

clips), a transition between each pair of consecutive video 40 and audio clips) are stored as digital data in a computer I 

and/or still image clips, and optionally also a sequence of memory, selected clips are displayed in elongated windows Y 

audio clips. Each of the transitions between video (and/or (known as "tracks") on a display screen, and editing opera- J 

still image) clips can be a simple "cut," in which the last tions are performed on the clips in response to manipulation I 

frame of a first clip is concatenated with the first frame of a of displayed cursors and icons to assemble and preview an / 

second clip. Alternatively, one or more of the transitions can 45 edited video program. 

be a more complicated (special effect) transition, such as a The preferred embodiment of the inventive apparatus is a 
wipe, fade, or dissolve. For example, an edited program can computer system programmed to display representations of 
consist of a first video clip, followed by a simple cut to a video, still image, and audio clips desired positions alo np 
second video clip, followed by a dissolve transition to a third a dis played time ruler, in tracks of a construction window , 
video clip. 50 The system dynamically generates each video clip to be 
Typically, an edited video program consists of a sequence displayed by retrieving from storage all frames of the video 
of video and still image clips (with special effect transitions clip (or every "Nth" frame of the stored clip in accordance 
between consecutive pairs of these clips), and two or more with a user-selected time compression factor), and display- 
sequences of audio clips (for example, two sequences of m § tDe retrieved frames. 

audio clips representing left and right channels of an audio 55 Animated "special effect" icons, each representing a spe- 
soundtrack). cial effect, are displayed in a separate special effects track 
It is conventional to define an edited video program by also oriented parallel to the time ruler. Each special effect 
employing a programmed computer to perform video editing icon can represent a special effect transition between two 
operations in which the computer processes stored digital clips (such as a dissolve, fade, and wipe), 
data representing segments of the video program. A con- 60 The user instructs the computer to assemble a video 
ventional system of this type is disclosed in U.S. Pat. No. program from the stored video clips (and optionally also 
4,538,188, issued Aug. 27, 1990, to Barker, et al. The system stored audio clips and clips representing still image images 
of U.S. Pat. No. 4,538,188 employs a computer to control or text) by arranging displayed clips and displayed special 
the display and manipulation of pictorial labels (or label effect icons in a desired sequence along the time ruler, 
pairs), each representing an unedited video segment (or 65 In a preferred embodiment, the computer system of the 
transition between consecutive video segments), in order to invention is programmed to perform the following opera- 
assemble an edited video program from unedited video tions: 
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1. select a new "in" point, "out" point, or both, for a FIG. 7 is a modified version of the FIG. 6 display, which 
displayed clip (in the edited video program) in response to is produced by dragging the right edge of the video clip of 
the positioning of a cursor at an edge of the displayed clip FIG. 6, in an embodiment of the invention. 

and manipulation of an input device to "drag" the edge along FIG. 8 is a portion of a construction window having a 

the time ruler, 5 single audio clip displayed in one audio track, generated in 

2. select a special effect transition between a pair of video an embodiment of the invention, 
clips displayed in first and second parallel tracks (and having 

overlapping portions along the time ruler), by displaying a FIG - 9 * 3 modlfied version of the FIG. 8 display, which 

selected transition icon in a third parallel track in alignment 15 P™duced by altering the level control icon of the FIG. 8 

with the overlapping portions of the video clips; 1Q display, in an embodiment of the invention. 

3. select special effect transition parameters (such as FIG. 10 is a construction window displayed by a preferred 
transition duration) by displaying a transition icon in a embodiment of the inventive apparatus, with a video clip 
special effects track (where the special effects track is and an overlay clip displayed in separate tracks, 
displayed below a first video track and above a second video FIG. 11 is a modified version of the FIG. 10 display, 
track, and where overlapping video clips are displayed in the wn ich is produced by altering the level control icon of the 
first and second video tracks), and positioning a cursor over p|Q display. 

the transition icon while manipulating an input device; t ~ . • i . .-, JL c , 

* , , . FIG. 12 is a special effects menu displayed by a preferred 

4. control superimposit.on of an over ay clip (representing cmbodiment of tne mventive ^ f nclu j in / a m of 
video, still .mage, or text) with a mam (video or ail image) ; ( ff transition icons. 

chp by displaying the clips in separate tracks, displaying a F 

level control icon in alignment with the overlay chp, and 20 FIG - 13 1S a M of S1X transition icon displays, represent- 

manipulating the level control icon to display a graph m 8 the animated dls P la Y °f a special effect transition icon, 

representing a time-varying weighting function for combin- generated in an embodiment of the invention, 

ing the main and overlay clips; FIG- 14 is a mosaic filter control menu displayed by a 

5. preview a video program defined by clips displayed in preferred embodiment of the invention. 

a desired sequence along a time ruler of a construction 25 FIG. 15 is a special effect transition icon of a type 
window by retrieving the clips from memory, processing the displayed by a preferred embodiment of the invention, 
retrieved clips (for example, in accordance with special 
effects transitions defined by transition icons displayed 
along the time ruler), and sequentially displaying the pro- 
cessed clips as a preview in a separate video window; With reference to FIG. 1, a preferred embodiment of the 

6. cache (in random access cache memory) all frames invention includes programmed microprocessor 31, main 
retrieved in response to a command to update a displayed memory 32 (a random access memory), mass storage device 
window or to preview a program defined by clips displayed 33, and interface 37, all connected along a system bus 34, 
in a desired sequence along a time ruler of a construction 35 and mouse 38, audio speaker 39, and computer keyboard 40 
window; connected to interface 37. 

7. process one or more selected video clips by filtering the Expansion bus 35 is connected to system bus 34, and 
corresponding digital video data with a temporally varying digital video board 30 and computer monitor 36 (having a 
mosaic filter (having user-selected time-varying filter display screen) are connected along expansion bus 35. 
characteristics); and 40 Microprocessor 31 runs the operating system and applica- 

8. perform special effects processing by filling the alpha lions software that controls the operation of digital video 
channel of a first clip with a first value (i.e., a hexadecimal board 30 (and the other system components). Preferably, this 
"00"), filling the alpha channel of a second clip with a software includes Apple QuickTime system extension soft- 
second value (i.e., a hexadecimal "IP*), performing a first ware available from Apple Computer, Inc. (or software 
special effect process in which selected pixels from the first 45 equivalent thereto) for integrating video and audio data into 
chp are combined with selected pixels of the second clip to Macintosh applications. 

generate a processed clip, and processing alpha channel data Appropriately programmed microprocessor 31 performs 

of the processed clip to identify one or more edges of the all necessary digital processing operations (including special 

processed clip (for example, to facilitate further special effects processing and filtering, including mosaic filtering to 

effect processing of the processed clip). 50 be described below) on data sets received from memory 32 

BRIEF DESCRIPTION OF THE DRAWINGS or 33, Doard 30 drives monitor 36 » 10 display (in windows 

on the monitor's screen) data transferred to bus 35 from 

FIG. 1 is a preferred embodiment of the inventive appa- system bus 34 after undergoing special effects processing, 

ratus * filtering, or other processing in microprocessor 31. 

FIG. 2 is a construction window displayed by a preferred 55 Preferably, board 30 processes 24-bit color video data for 

embodiment of the inventive apparatus. display on monitor 36, and microprocessor 31 has 32-bit 

FIG. 3 is a construction window (of the FIG. 2 type) with addressing capability identical or equivalent to that of the 

two video clips, one still image chp, one audio clip, and two Apple System 7 product available from Apple Computer, 

special effect transition icons displayed in separate tracks Inc. 

thereof. 60 M&in memory 32 stores the program and data sets for 
FIG. 4 is a project window displayed by a preferred microprocessor 31, and preferably has at least eight mega- 
embodiment of the inventive apparatus, byte capacity. Microprocessor 31 and main memory 32 are 
FIG. 5 is another project window displayed by a preferred preferably implemented as a Macintosh™ II (or later model) 
embodiment of the inventive apparatus. computer having at least eight megabytes of random-access 

FIG. 6 is a portion of a construction window having a 65 memory, 

single video clip displayed in one video track, generated in Mass storage device 33 is typically a hard disk (preferably 

an embodiment of the invention. having at least 80 megabyte capacity), but can alternatively 
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be a magneto-optical disk or other read/write device, or (in enlargement of the project window, the system will display, 

some embodiments) a read-only device such as a CD ROM. side-by-side, additional copies of the thumbnail of each still 

Microprocessor 31 and mass storage device 33 commu- image clip. For example, in the enlarged project window of 

nicate with each other (and with main memory 32) over FIG. 5, four copies of still image clip thumbnail 56 are 

system bus 34. Bus 34 also allows data communication to 5 displayed. 

expansion bus 35. In the Apple Macintosh™ II family of Preferably, the user can control the time scale on which 

personal computers, expansion bus 35 adheres to the NuBus clips are displayed in the project window (i.e., the degree of 

standard. Additional information on NuBus operation is time compression with which a video clip is displayed in the 

available from Texas Instruments, Dallas, Tex. project window). This can be done by positioning cursor 49 

Additional details regarding preferred embodiments of 10 on a "time unit" icon 58a in time compression window 58 

hardware (and software) for implementing the invention are and dragging icon 58a to a different station within windo\ij 

set forth in the "Adobe Premiere™ User Guide- Macin- 58 (by manipulating mouse 38). Preferably, icon SSa can b^ 

tosh™ Version," dated September 1991, available from positioned at any of several stations, each representing i 

Adobe Systems Incorporated, which document is incorpo- different time scale (and time compression factor . 

rated herein by reference. 15 Examples of such stations include: "single frame" (for 

Microprocessor 31 is programmed with editing software displaying each frame of a video clip, i.e., one frame for 

and user interface software (to be described below). The user each 1/30 second segment of a video clip representing vide* ) 

interface software accesses the editing software (and the data having a frame rate of 30 per second), "two frames' 

other system software) in response to commands entered by ( one displayed frame for each two frames of a video clip; , 

the user using mouse 38 or keyboard 40, and is capable of 20 " one second" (one displayed frame for each segment of 1 

instructing board 30 to generate displays of the type shown vide0 cli P having 1 second duration), and "two minutes " 

in FIGS. 2-14 on computer monitor 36. (one displayed frame for each segment of a video clip having 

Digital data representing video clips, still image clips, and 2 mmute duratl0n )- 

audio clips are stored in mass storage device 33. Micropro- In general, the system responds to any command requir- 

cessor 31 "caches" selected portions of this data, including ing display of a video clip in a window (which can be a 

frames of data selected for display on monitor 36, by preview window, or a construction window of the type to be 

copying the data into random access memory 32 (which will described below) by retrieving from storage, caching, and 

sometimes be denoted herein as a "cache" memory). displaying every "Nth" frame of the selected video clip. The 

.When a user initiates an editing project, the system opens, 30 factor " N " * determined by the currently selected time 

/Vnd displays on monitor 36, a project window (such as tha t compression factor. For example, if the user has selected the 

/ shown in FIG. 4 or FIG. 5) and a construction window (such above-described one second time compression factor, the 

as that shown in FIG. 2). By manipulating keyboard 40 f actor * ^ uals 3 °> ™d the system will display every 30th 

\ and/or mouse 38, the user selects stored virion nip* ftnd frame of a selected video clip (if the clip has a frame rate of 

/ optionally a lso still image clips and audio clips) from mass 35 30 frames per second). 

/ stora^jie*dce»3a. In response to selection of each clip, For another example, in response to selection of a stored 

/ programmed microprocessor 31 generat es an imag e video clip from mass storage device 33 for inclusion in a 

^ ("thumbnajOJpjUhe clip, and displays the thumbnail (and project (and display in the project window) with time unit 

corresponding identifying text) in the project window. FIG. icon 58a at a "single frame" station in time compression 

4 shows four such thumbnails 50, 52, 54, and 56 displayed 4 0 window 58, the system will display in the project window as 

in a project window. FIG. 4 also shows identifying text 51 many frames of the clip as will fit in the currently configured 

for a video clip having thumbnail 50, identifying text 53 for project window, and will cache all the displayed frames in 

a video clip having thumbnail 52 (and a corresponding audio cache memory 32. If the user then enlarges the project 

clip having thumbnail 54), and identifying text 55 for a still window, the system will retrieve a larger subset of the clip's 

image clip having thumbnail 56. The identifying text pref- 45 frames from cache memory 32 (and mass storage memory 

erably includes the size in pixels (i.e., 320 pixelsx240 33), display this larger subset of frames in the enlarged 

pixels) of each video or still image clip, a qd the clip duration project window, and cache the displayed frames in cache 

(in the S MPTE format "hours: minutes: seconds: frames"). memory 32. 

V^The thumbnail for a video clip is (in a preferred For another example, in response to selection of a stored 

embodiment) one (possibly reduced), displayed frame 50 video clip from mass storage device 33 for inclusion in a 

thereof, such as thumbnail 50 and 52 in FIG. 4. For a still project with time unit icon 58a at a "two frames" station in 

image clip, the thumbnail is (in a preferred embodiment) the window 58, the system will display in the project window as 

corresponding image (or text), such as thumbnail 56 in FIG, many frames of the time-compressed clip as will fit in the 

4. For an audio clip, the thumbnail is (in a preferred currently configured project window (i.e., only the first, 

embodiment) a graph of a corresponding audio signal 55 mird, and fifth frames, if only three frames will fit in the 

waveform, such as thumbnail 56 in FIG. 4. project window), and will cache the displayed frames in 

Preferably, the system is programmed to enable a user to cache memory 32. If the user then enlarges the project 

enlarge the project window by manipulating mouse 38 to window, the system will retrieve a larger subset of the 

position cursor 49 on size icon 57, and then dragging size time-compressed clip's frames from cache memory 32 (and 

icon 57 toward the right (on the screen of monitor 36). In 60 mass storage memory 33), display this larger subset of 

response to enlargement of the project window, the system frames in the enlarged project window, and cache the 

will display thumbnails representing additional frames of displayed frames in cache memory 32. 

each previously selected video clip (each of which already To generate an edited video program, the user drags 

has a thumbnail displayed in the project window). For thumbnails of selected clips from the project window to a 

example, FIG. 5 shows an enlarged version of the FIG. 4 65 construction window of the type shown in FIG. 2 (by 

project wi ndow T with three additional frames (50 A. SOB, and manipulating mouse 38), and moves the thumbnails into a 

50Q of the video clip having thumbnail 50. In response to desired arrangement in the construction window. The con- 
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struction window displays a time ruler 60 in a timeline will also display all frames of the time-compressed clip in 

window along its top edge, and elongated windows one track of the construction window (with one displayed 

("tracks") oriented parallel to time ruler 60. FIG. 2 shows frame per "one second" interval along time ruler 60), and 

seven tracks, labeled "A," "Fx," "B," "Super," "A'," "B\" will display a version of time ruler 60 scaled for the selected 

and "C," respectively. 5 "one second" time scale (having small tick marks represent - 

Because the thumbnails displayed in tracks of the con- ing frames between each pair of large tick marks, with the 

struction window represent entire video clips or time- distance between each pair of large tick marks representing 

compressed entire video clips, the following description of one second). 

the construction window (with reference to FIGS. 2-3 and A very simple edited video program is defined by video 

6-11) will refer to thumbnails displayed in tracks of the 10 (and/or still image) clips arranged side-by-side in a single 

construction window as "clips." track (i.e., track A), with the "Out" point (right edge) of each 

lime ruler 60 represents the timeline of the edited video clip butting against the "In" point (left edge) of the next clip, 

program. The user can expand the construction window by anci wi th no audio clips displayed in audio tracks A\ B\ and 

dragging size icon 57 (in the same manner described above C\ In response to entry of a "preview" command with this 

with reference to identical icon 57 of the project window). 15 arrangement of clips displayed in the construction window, 

Because a complete edited program may have duration the system previews the program by sequentially displaying 

longer than can be represented by clips displayed on a single ( in a preview window) the clips that are simultaneously 

monitor screen, the system permits the user to display any displayed in the construction window, in a time order 

selected portion of program in the construction window. For corresponding to the spatial position of the clips along time 

example, the user can instruct the system to display the clips 20 60. 

for a first portion of the program in a construction window To generate more complicated video programs (with 

(with the construction window displaying a first portion of overlapping clips), the user displays pairs of overlapping 

time ruler 60 extending from the program start time 0:00:00 clips in different tracks of the construction window (as 

to intermediate time M:MM:MM), and thereafter instruct shown in FIG. 3). 

the system to display the clips for the remaining portion of 25 With reference to FIG. 3, the user can specify a special 

the program in the construction window (with the construe- effects transition between video clip 62 and still image clip 

lion window displaying the remaining portion of time ruler 63 by positioning clip 62 in track A and clip 63 in track B, 

60 extending from time M:MM:MM to a program end time and positioning a special effect transition icon 65 in a track 

N:NN:NN). 3Q FX. The user adjusts "In" and "Out" points of icon 65 (the 

When clips have been arranged in the construction win- left and right edges of icon 65, respectively) so that icon 65 
dow along time rule 60, the user can enter a "preview" coincides with the overlapping portions of clips 62 and 63 
command to microprocessor 31 to cause the system to along time ruler 60. Because icon 65 represents a cross- 
generate a video program defined by the displayed clips. The dissolve transition (to be described below), the FIG. 3 
manner in which the system responds to a "preview com- 35 display specifies a video program including a cross dissolve 
mand" will be more fully described below, but can be transition from clip 62 to clip 63, beginning at the "In" point 
summarized as follows: the system retrieves from mass of icon 65 and ending at the "Out" point of icon 65. 
storage memory 33 (or cache memory 32) all data corre- FIG. 3 also shows a "push" special effect transition icon 
sponding to the displayed clips, stores the retrieved frames 66 in track FX, in alignment with overlapping portions of 
in cache memory 32, and displays the cached frames sequen- 4Q video clip 64 (in track A) and still image clip 63 (in track B). 
tially (as full-motion video) in a window on the screen of Because icon 66 represents a push transition (to be described 
monitor 36 (and optionally also plays the corresponding below), the FIG. 3 display specifies a video program includ- 
audio soundtrack channels on speakers). ing a "push" transition from clip 63 to clip 64, beginning at 

In the preferred embodiment shown in FIG. 2, the con- the "In" point of icon 66 and ending at the "Out" point of 

struction window has seven tracks, all oriented parallel to 45 icon 66. 

time ruler 60. Only audio clips can be displayed in the FIG. 3 also shows audio clip 67 in audio track A', which 
bottom three tracks (identified as tracks A', B', and C), and represents a single-channel (monaural) audio soundtrack, 
only video or still image clips can be displayed in the top Because the "In" point (left edge) of audio clip 67 is aligned 
four tracks (identified as tracks A, FX, B, and "Super"). with the "In" point of clip 63, the soundtrack for the program 
The large tick marks of time ruler 60 represent the time 50 will commence simultaneously with the start of the cross- 
scale for the video program (with time displayed in SMPTE dissolve from clip 62 to clip 63, and the soundtrack will 
format), and the small tick marks represent frames or continue during display of clips 63 and 64. Optionally, the 
seconds (depending on the currently selected time scale). user could add second and third audio clips to tracks B* and 
The user selects a lime scale by manipulating icon 58a in C to specify a three-channel audio soundtrack (which might 
window 58 (shown in FIG. 2) in the same manner as 55 correspond to left and right stereo channels, and a "sur- 
described above with reference to identical icon 58a and round" or "sub-woofer" channel). 

window 58 of the project window of FIG. 4. As described The system is programmed so that manipulation of audio 

above with reference to FIG. 4, icon 58a preferably can be level control icon 67A (displayed with clip 67 in track A') 

positioned at any of several stations, each representing a allows the user to adjust the level of the corresponding audio 

different time scale (including, for example, "single frame," 60 track, in the same way that sound is mixed in record and 

"two frames," "one second," and "two minutes" stations). television production. With reference to FIGS. 8 and 9, the 

For example, if the user drags a video clip into the con- user positions a pointer (cursor) 68 on middle line 69 of icon 

struction window with time unit icon 58a positioned at a 67A (by manipulating mouse 38). The user then "clicks" 

"one second" station in window 58 (as shown in FIG. 3), the mouse 38 (activates a control on mouse 38) to create a 

system will cache (in cache memory 32) a time-compressed 65 handle 71 (a black dot), and drags 71 handle up or down 

version of the video clip (i.e., it will cache one frame of the relative to line 69 to program the system to execute an audio 

clip for each one-second segment of the clip). The system clip fade in or fade out, respectively, and to deform a 
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displayed "rubber-band" level control line 70. The default 
position of rubber-band line 70 is a position coincident with 
middle line 69, which represents a mid-range audio level for 
clip 67. Line 70 has "rubber-band" characteristics in the 
sense that, in response to a user's dragging of a handle 71 5 
vertically at one position along time ruler 60, the system 
automatically deforms line 70 along part of its length in 
accordance with a model in which the portion of line 70 to 
the left of the handle simulates an elastic band (to provide 
for smooth temporal variations in audio level assigned to 10 
consecutive segments of audio clip 67). Thus, if the user 
drags a handle 71 downward from the position of pointer 68 
in FIG. 9, the system automatically moves downward a 
portion of line 70 on the left side of pointer 68, as if that 
portion of line 70 were made of elastic material. is 

The user can create as many handles 71 as desired (three 
handles 71 are indicated in FIG. 9). Each ascending portion 
of line 70 represents a fade in, and each descending portion 
of trace 70 represents a fade out. In response to entry of a 
preview command, the system retrieves from storage the 20 
audio data for audio clip 67 and plays the audio clip through 
a speaker (such as speaker 39 of FIG. 1), with an instanta- 
neous volume level determined by the position of line 70 
relative to line 69 at the corresponding position along time 
ruler 60. 25 

Next, with reference to FIGS. 2, 10, and 11, another way 
will be described in which a user can control superimposi- 
tion of overlapping video (or still frame image) clips dis- 
played in the construction window. To accomplish this, the 
system is programmed to superimpose a clip displayed in the 30 
"Superimpose" track (identified by label "Super" in FIGS. 2, 
10, and 11) with a clip displayed (along the same segment 
of time ruler 60) in track A or B, or with overlapping clips 
displayed in tracks A and B. When the system generates and 
previews the corresponding program, the images corre- 35 
sponding to the clips in tracks A or B (or both A and B) 
shows through transparent parts of the superimposed image. 

To superimpose a video or still image clip (referred to as 
an "overlay" clip) with a clip (referred to as a "main" clip) 4Q 
displayed in track A, B, or both A and B, the user drags the 
overlay clip to the "Super" track, and aligns it with the 
relevant portion of the clip in track A. For example, FIGS. 
10 and 11 show a main clip 70 in track A and an overlay clip 
72 in a "Super" track. The construction window of FIGS. 10 45 
and 11 has been reconfigured to delete tracks B and FX. In 
a preferred embodiment, the user can reconfigure the con- 
struction window of FIGS. 10 and 11 to add tracks B and FX 
of the type shown in FIG. 2. 

The user specifies which portions of the overlay clip are 50 
transparent, in one of two ways. In one alternative, the user 
specifies a color (or range of colors) of the overlay clip as 
being transparent (so that the main clip will show through 
portions of the overlay clip having the specified color or 
range of colors). This can be done by displaying a transpar- 55 
ency setting menu having icons which can be manipulated 
using mouse 38 to specify the "transparent" color or range. 
In another alternative, the overlay clip includes a special 
"alpha" channel, comprising bits which define a grey scale 
level for each pixel of each frame of the overlay clip. In this 60 
case, the system can be programmed to interpret the grey 
scale levels as degrees of transparency. An example of a clip 
having such an alpha channel is a clip consisting of 32-bit, 
still image frames generated by the "Photoshop" software 
commercially available from Adobe Systems Incorporated. 65 

It is often desirable to generate an edited program having 
a title (or other overlay image) which fades in or out, or has 
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a semi-translucent appearance, when superimposed on a 
main image. In a preferred embodiment, the system of the 
invention is programmed to enable the user to achieve such 
effects in a convenient, intuitive manner. This is accom- 
plished by manipulating a level control icon, such as fade 
control bar 74 displayed in the "Super" track of FIGS. 10 
and 11. To adjust how "strongly" the overlay image will be 
displayed on the main image (or in other words, to adjust the 
weight with which overlay clip 72 is combined with main 
clip 73), the user manipulates mouse 38 to position a pointer 
76 on top (horizontal) line 74A of fade control bar 74. The 
user then "clicks" mouse 38 to display a handle 76 A (a black 
dot), and drags handle 76A up or down within fade control 
bar 74 to deform a displayed "rubber-band" level control 
line 78. The default position of rubber-band line 78 is a 
position coincident with top line 74A, which represents 
maximum weight for overlay clip 72. If the user drags 
handle 76A downward (upward), the system assigns 
decreased (increased) weight to the corresponding segment 
of the overlay clip. With handle 76A at the bottom of icon 
74, the corresponding overlay image will be completely 
non-visible when superimposed on the corresponding seg- 
ment of main clip 73. 

Line 78 has "rubber-band" characteristics in the sense 
that, in response to a user's dragging of handle 76A verti- 
cally at one position along time ruler 60, the system auto- 
matically deforms line 78 along part of its length in accor- 
dance with a model in which the portion of line 78 to the left 
of the handle simulates an elastic band (to provide for 
smooth temporal variations in the weight assigned to con- 
secutive segments of overlay clip 72). Thus, if the user drags 
handle 76A downward from the position shown in FIG. 11, 
the system automatically moves downward a portion of line 
78 on the left side of handle 76A, as if that portion of line 
78 were made of elastic material. 

One advantage of the invention is that it provides a 
convenient and intuitive way for a user to select "In" and 
"Out" points for each clip to be included in an edited video 
program. In contrast, prior art editing systems accomplish 
this function in an inconvenient manner, in which the user 
views sequentially displayed video frames, freezes the 
sequential display at a desired frame, and marks the frame 
as an "In" or "Out" point of a video segment. This prior art 
technique makes it very difficult for a user to perform many 
routine editing operations, such as extending a video seg- 
ment to match certain actions in another video segment (or 
to match a desired portion of an audio soundtrack). 

The inventive system is preferably programmed to enable 
a user to select "In" and "Out" points for each clip of a video 
program, in the manner to be described next with reference 
to FIGS. 6 and 7. To align a clip along time ruler 60, the clip 
is displayed in a track of the construction window, and the 
left or right edge of the clip is dragged along the track to a 
desired "In" or "Out" point along the time ruler 60. The 
precision with which each edge can be positioned along time 
ruler 60 depends on the currently selected time scale for the 
construction window (the time scale is selected, in the 
manner explained above, by manipulating icon 58). To align 
clips with maximum precision, the time unit should be 
selected to have its lowest value (i.e., icon 58 configured in 
the "single frame" setting, in which each frame of the clip 
is displayed in the construction window). _ 

As shown in FIG. 6, left edge 81 of clip 80 is aligned at 
SMPTE time code 0:00:10:00 of time ruler 60, indicating 
that clip 80 commences (i.e., has an "In" point) at the tenth 
second of the video program being edited. Right edge 82 of 
clip 80 is aligned at SMPTE time code 0:00:11:00 of time 
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ruler 60, indicating that clip 80 ends at the eleventh second a "Dither Dissolve" icon, representing a transition in 

of the v ideo pro grain. To maintain this "In" point for clip 80, which image A dissolves into image B; 

and insert a lon^duration version of clip 80 into the video a "Funnel" icon, representing a transition in which the A 

program, the user positions a "stretch" cursor 83 at right { ^ Ued lnf fa a revea]i tfae B • 

edge 82 of clip 80 (as shown in FIG. 6). The user then 5 U1 „ . . . . f , . L 

"grabs" edge 82 by manipulating a control on mouse 58 and an Inset 1 1C0D > ^presenting a transition in which a corner 

drags edge 82 toward the right along time ruler 60. When wl P e reveals ima S e B under ima 8 e * 

edge reaches the desired new "Out" point along ruler 60 an "Iris Round" icon, representing a transition in which a 

(i.e., the point marked with SMPTE time code 0:00:12:00, as circular wipe opens to reveal image B under image A; 

shown in FIG. 7), the user releases the control on mouse 38. 1Q an "Iris Square" icon, representing a transition in which a 

In response to this sequence of operations, the system edits rectangular wipe opens to reveal image B under image A; 

the video program to include a longer version of clip 80 a „ p Tum „ ^ represenling a lransition in which 

(having the original In point and he new Out point) and { A cm]s [Q reveal ; fi underaeatn 

retrieves from memory additional frames of clip 80 for <iw™ • ■ 

display in the construction window (i.e., the two new frames a PICT Mask lcon > representing a transition in which a 

of cli n 80 shown in FIG. 7, but not FIG. 6). 15 user selected 1-bit (black and white) image, of the conven- 

If the user enters the preview command with video clip 80 tio ° al PICr tv P e » and the s y stem re P laces the black in the 

displayed as shown in FIG. 6, the system would display (in PICT image with image A and the white in the PICT image 

a preview window) all frames of a one-second video seg- w * tn image B; 

ment. If the user enters the preview command with video a "Push" icon, representing a transition in which image B 

clip 80 displayed as shown in FIG. 7, the system would 20 appears to push aside image A; and 

display (in the preview window) all frames of a two-second a "Radial Wipe" icon, representing a transition in which 

video segment (with the first one -second portion of such a i me fixed at one corner of the screen sweeps across image 

two-second segment identical to the segment referred to in A, revealing image B. 

the previous sentence). In response t0 a command from the user to activate the 

Similarly, the user can set a new "In" point for a clip by 25 spedal effects mtmj ^ the system animates each of the 

dragging the clip's left edge, along a track in the construe- transition icons displayed in the special effects menu. For 

tion window, with the stretch icon. example, FIG. 13 is a set of six transition icon displays, 

Also similarly, microprocessor 31 is preferably pro- representing the animated display of the "Push" transition 

grammed to enable a user to split a clip into two new clips icon of FIG. 12. The left-most image of FIG. 12 is displayed 

(which can be independently resized and repositioned along 30 at a first time (t=0), followed (in sequence) by the second 

time ruler 60) by positioning a "razor blade" cursor at image from the left, the third image from the left, the fourth 

desired position along the time ruler and manipulating a image from the left, the fifth image from the left, and the 

control on mouse 38 to define this desired position as the right-most image. After the right-most image is displayed, 

"Out" point of the first new clip and the "In" point of the the animation process repeats, and all six images are sequen- 

second new clip. 35 tially displayed again. 

Microprocessor 31 is preferably programmed to enable a The user includes a special effect transition in the program 
user to select a special effect transition icon, in the following being edited by selecting a corresponding transition icon 
manner, for display in the above -described "FX" track of the f r0 m the special effects menu, and dragging the selected 
construction window. In response to a command from the ^ transition icon to a desired position along the FX track of the 
user, microprocessor 31 causes a special effects menu (such construction window. As described above with reference to 
as the menu shown in FIG. 12) to be displayed in a window p(G. 3, the user adjusts "in" and "out" points of the 
on monitor 36. The special effects menu includes a set of transition icon in the FX track so that the left edge ("in" 
transition icons, each which can be selected by positioning point) and right edge ("out" point) of the transition icon are 
a cursor thereon using a mouse. ^ aligned with beginning and end points of a selected over- 
Each transition icon represents a different special effect lapping portion of a first clip in track A and a second clip in 
transition between two clips: an "A" clip (displayed in track track B (as transition icon 65 is configured in FIG. 3). In one 
A of the construction window, and associated with an "A" class of embodiments, the user adjusts in and out points of 
image); and a "B" clip (displayed in track B of the con- each transition icon by positioning a cursor on the left or 
struction window, and associated with a "B" image). The 5q right edge of the icon, and dragging the edge relative to time 
following fourteen transition icons are shown in FIG. 12 ruler 60, in the same manner (described above with refer- 
(more than or less than fourteen transition icons can be ence to FIGS. 6 and 7) that a user adjusts "in" and "out" 
displayed in alternative embodiments of the invention): points of a video or still image clip in track A or track B. 

a "Band Slide" icon, representing a transition in which the In a class of preferred embodiments, each transition icon 

B image appears to slide over the A image in horizontal or 55 includes a set of control icons for setting parameters of the 

vertical bars; corresponding special effect transition (such as the in point, 

a "Band Wipe" icon, representing a transition in which the out point, and duration of the transition). Icon 90 of FIG. 15 

B image is revealed under the A image by horizontal or is an example of a transition icon having such control icons, 

vertical bars; In FIG. 15, the control icon displayed in area 80 deter- 

a "Barn Doors" icon, representing a transition in which 60 mines the type of the special effect transition. The user 

the B image is revealed under the A image from the center determines the type of the transition by dragging a selected 

outwards; icon from a special effects menu (which can be the menu 

a "Checkerboard" icon, representing a transition in which described above with reference to FIG, 12) to area 80. 

boxes wipe in a spiral to reveal image B under image A; The user can specify a direction of movement for certain 

a "Cross Dissolve" icon (also shown within icon 65 in 65 transitions (i.e., the direction on the screen in which one clip 

FIG. 3), representing a transition in which image A fades "pushes" the other in a "push" transition) by "clicking" 

into image B; mouse 38 to select a desired one of edge control icons 82. 
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If there are at least two available directions for a transition, 
the corresponding icon 90 displays one edge control icon 82 
for each available position. For example, for a "push" 
transition (when a "push" icon of the type described with 
reference to FIG. 13 is displayed in area 80 of icon 90), eight 
edge control icons 82 may be displayed (as shown in FIG. 
15), each representing a different direction in which one of 
clips A and B will push the other during the transition. 

Icon 90 optionally includes track selection control icon 
88. The user can specify which clip (as between a clip in 
track A and an overlapping clip in track B) should play a 
given role in the transition, by clicking on icon 88 using 
mouse 38. 

Icon 90 optionally also includes forward/reverse control 
icon 84, and anti-aliasing control icon 86. The user can 
specify a forward or reverse direction for a transition by 
clicking on icon 84 using mouse 38. The user can set an 
anti-aliasing level (i.e., to a "low," "high," or "off*' level) by 
clicking on icon 86 using mouse 38. 

An additional feature of a preferred embodiment of the 
invention will be described next with reference to FIG. 14. 
In this embodiment, microprocessor 31 is programmed to 
perform any of a variety of filtering operations on selected 
clips. The user specifies each filtering operation by display- 
ing a corresponding filter control menu, and setting filtering 
parameters by manipulating filter control icons displayed on 
the filter control menu using mouse 38 and/or keyboard 40. 

An example of such a filter control menu (for a "mosaic" 
filter) is shown in FIG. 14, A mosaic filter divides the frames 
of the displayed clip into a grid of squares, and makes each 
square the average color of all the pixels within the square. 
The user can set the effect of the mosaic filter to gradually 
increase or decrease as the clip plays by adjusting "Start" 
and "End" control icons displayed in the mosaic filter 
control menu. By lengthening "Start" control icon 92, the 
user increases the level of the mosaic filter at the start of the 
clip (i.e., increases the size of the squares into which the first 
frame of the clip is divided). Similarly, by lengthening 
"End" control icon 94, the user increases the level of the 
mosaic filter at the end of the clip (i.e., increases the size of 
the squares into which the last frame of the clip is divided). 
The system is preferably programmed to interpolate the 
level of the temporally varying mosaic filter for intermediate 
frames of the clip. When the user has defined a temporally 
varying mosaic filter for a clip, and then enters the preview 
command, the system filters the digital data corresponding 
to each frame of the clip (in accordance with the defined 
mosaic filter) before displaying the filtered clip in the 
preview window. 

Another feature of a preferred embodiment of the inven- 
tion will be described next. This feature contemplates that 
microprocessor 31 can be programmed to execute any of a 
variety of special effects on selected clips. One preferred 
way to accomplish this is to program microprocessor 31 
with "plug-in" special effect software modules, each of 
which, performs a phase of a more complicated special 
effect. This feature also contemplates that the frames of 
digital data to be processed include an alpha channel (i.e., an 
8-bit, transparency-scale representation of an image for each 
frame) associated with color channels (which can comprise 
three 8-bit color values for each pixel of the frame). 

When performing special effects processing (requiring 
combination of two clips), the programmed microprocessor 
31 of the invention preferably fills the alpha channel of the 
first clip with a first value (i.e., a hexadecimal "00"), fills the 
alpha channel of the second clip with a second value (i.e., a 
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hexadecimal "IF"), and then performs a first special effect 
process (i.e., in a plug-in special effect software module) in 
which selected pixels from the first clip are combined with 
selected pixels of the second clip to generate a processed 

5 clip. Then, when the processed clip undergoes subsequent 
processing, the alpha channel data of the processed clip is 
examined to identify one or more edges of the processed 
clip. For a given pixel, if the pixel's alpha channel matches 
the alpha channels in the four adjacent pixels (the pixels left, 

10 right, above, and below it) then there is no edge at that pixel. 
If, however, one or more of the alpha channels differs, an 
edge is determined to exist at that pixel (and any desired 
operation can be applied at that location, such as colored 
edge or a smoothing operation). 

35 This technique allows the second plug-in special effect 
software module (or other software) to identify edges of the 
processed clip without the need to specifically generate an 
edge region data structure. 

The choice of "00" and "IF" values allows for an addi- 

20 tional speed increase in the comparison. Instead of compari- 
son of the alpha channels as described above, the alpha 
channel values of the five neighboring pixels can be added 
together as byte sized values, A result of zero or a negative 
value indicates no edge, while any other value indicates an 

25 edge. 

Next, the caching steps (briefly described above) imple- 
mented by a preferred embodiment of the invention will be 
more fully described. Data are written into the cache 

3Q memory (i.e., random access memory 32 of FIG. 15) in 
response to each user command for updating the construc- 
tion window, or for previewing a program defined by a 
currently displayed construction window. An example of the 
latter type of command is a command to change the time 

35 scale of time ruler 60. 

In response to each such command, the system calculates 
the addresses of the data that must be retrieved to update the 
construction window (or a preview window for previewing 
the program). The system then searches the cache memory 

40 to retrieve therefrom any of the required data currently 
stored in the cache memory. After retrieving any of the 
required data currently stored in the cache memory, the 
system retrieves from its mass storage memory (memory 33 
of FIG. 1) the remaining required data. The system then 

45 caches all the retrieved data in the cache memory, processes 
the retrieved data in response to the command, and displays 
the processed data to update the screen display. As a result 
of the caching step, the system reduces the memory access 
time required for responding to future construction window 

50 updating or preview commands. 

For example, in an embodiment of the invention, in 
response to a command for previewing a program, micro- 
processor 31 retrieves all frames of the program (from cache 
memory or mass storage memory), processes the retrieved 

55 frames in accordance with special effects transitions defined - 
by transition icons currently positioned in the construction 
window (and in accordance with any user-specified filtering 
operation), and causes the processed frames to be sequen- 
tially displayed in a preview window on the screen of 

60 monitor 36. The system also caches the retrieved frames (in 
their unprocessed form). 

In preferred embodiment, microprocessor 31 "com- 
presses" each frame to be cached in memory 32, in the sense 
that it causes only some of the bits of the frame to be cached. 

65 For example, if a clip comprises 32-bit digital video data, 
including an eight-bit alpha channel for each frame of the 
clip, retrieved frames of the clip may be "compressed" by 
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eliminating the alpha channel before caching the remaining 
24-bit digital video data defining each frame. Such an alpha 
channel can be employed by microprocessor 31 to perform 
certain special effects processing (described above), and 
accordingly, such special effects processing must be per- 
formed on the retrieved frames before the frames are "com- 
pressed." 

Various other modifications and alterations in the struc- 
ture and method of operation of this invention will be 
apparent to those skilled in the art without departing from 
the scope and spirit of this invention. Although the invention 
has been described in connection with specific preferred 
embodiments, it should be understood that the invention as 
claimed should not be unduly limited to such specific 
embodiments. 

What is claimed is: 

1. A method for editing a video program which includes 
a sequence of clips and transitions between the clips, the 
method comprising the steps of: 

(a) displaying a time line; 

(b) generating a display representing the video program 
clips by digitizing and thereafter simultaneously dis- 
playing a frame from each of a plurality of the digitized 
clips on tracks along and in a direct time relationship to 
the time line the width of the displayed frames repre- 
senting the time duration of the clip represented by that 
frame, together with transition icons, the transition 
icons representing the transitions between the plurality 
of the digitized clips; 

(c) editing the video program by modifying the display. 

2. The method of claim 1, wherein step (c) includes the 
step of: 

dynamically generating and displaying the frames in 
digital form in response to commands for modifying 
the display. 

3. The method of claim 2, wherein each of the frames has 
a left edge aligned at an in point along the time line and a 
right edge aligned at an out point along the time line, and 
wherein step (c) includes the step of: 

changing an in point of a first one of the clips by changing 
a position along the time line of the left edge of the 
frame representing the first one of the clips. 

4. The method of claim 2, wherein each of the has a left 
edge aligned at an in point along the time line and a right 
edge aligned at an out point along the time line, and wherein 
step (c) includes the step of: 

changing an out point of one of the clips by changing a 
position along the time line of the right edge of the 
frame representing one of the clips. 

5. The method of claim 1, wherein a first one of the 
transition icons is displayed in a special effects track, and 
has a left edge aligned at an in point along the time line and 
a right edge aligned at an out point along the time line, and 
wherein step (c) includes the step of: 

changing the in point of the first one of the transition icons 
by changing a position along the time line of the left 
edge of said first one of the transition icons. 

6. The method of claim 1, wherein a first one of the 
transition icons is displayed in a special effects track, and 
has a left edge aligned at an in point along the time line and 
a right edge aligned at an out point along the time line, and 
wherein step (c) includes the step of: 

changing the out point of the first one of the transition 
icons by changing a position along the time line of the 
right edge of said first one of the transition icons. 

7. The method of claim 1, wherein step (c) is performed 
in response to a command for modifying the display, and 
wherein step (c) includes operations of: 
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(d) identifying data needed to modify the display in 
response to the command; 

(e) retrieving from a cache memory at least a first portion 
of the data currently stored in the cache memory; 

5 (f) after step (e), retrieving any remaining portion of the 

data from a mass storage memory; 
(g) processing the data retrieved during steps (e) and (0 

to generate processed data for modifying the display in 

response to the command; and 
io (h) caching the data retrieved during steps (e) and (f) in 

the cache memory by compressing the data and caching 

the resulting compressed data in the cache memory. 

8. The method of claim 1, wherein the display includes a 
first frame in digital form displayed on a first track, a second 

15 frame in digital form displayed on a second track with an 
overlap portion of the second frame aligned along the time 
line with an overlap portion of the first frame, and a 
transition icon displayed in a third track in alignment with 
the overlap portion of the second frame. 

9. The method of claim 8, wherein the transition icon has 
20 an associated control icon, and wherein step (c) includes a 

step of: 

modifying a parameter of a transition in the video pro- 
gram between the first clip and the second clip in 
response to manipulation of the control icon using an 
25 input device. 

10. The method of claim 8, wherein the transition icon is 
an animated icon which simulates a transistion in the video 
program between the first clip and the second clip. 

11. The apparatus of claim 8, wherein the transition icon 
30 is an animated icon which simulates a transition in the video 

program between the first clip and the second clip. 

12. The method of claim 1, wherein step (c) adds to the 
video program a special effect transition between a first clip 
and a second clip, and wherein step (c) includes the steps of: 

(d) displaying a first frame on a first track, and a second 
frame on a second track with an overlap portion of the 
second frame aligned along the time line with an 
overlap portion of the first frame; 
40 (e) selecting a first transition icon; and 

(f) positioning the first transition icon in a third track in 
alignment with the overlap portion of the second frame. 

13. The method of claim 12, wherein step (e) includes the 
steps of displaying a menu of transition icons including the 

45 first transition icon and selecting the first transition icon in 
response to manipulation of an input device, and wherein 
step (f) includes a step of dragging the first transition icon 
from the menu to the third track in response to manipulation 
of the input device. 

5 q 14. The method of claim 13, wherein each of the transition 
icons of the menu is an animated icon which simulates an 
available transition in the video program. 

15. The method of claim 1, wherein step (c) includes the 
steps of: 

55 displaying a filter menu having control icons representing 
parameters of a temporally varying filter for filtering a 
selected one of the clips; and 
determining the parameters of the temporally varying 
filter in response to manipulation of the control icons 

60 using an input device. 

16. The method of claim 15, wherein step (c) includes the 
step of: 

filtering the selected clip with the temporally varying 
filter. 

65 17. The method of claim 15, wherein the temporally 
varying filter is a mosaic filter, and the parameters include a 
start level and a final level for the mosaic filter. 
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18. The method of claim 1, wherein step (c) includes a 
step of performing special effects processing on a first clip 
and a second clip, said step of performing special effects 
processing including the steps of: 

filling an alpha channel of the first clip with a first value; 
filling an alpha channel of the second clip with a second 
value; 

performing a first special effect process in which selected 
data of the first clip are combined with selected data of 
the second clip to generate a processed clip having an 
alpha channel; and 

processing data in the alpha channel of the processed clip 
to identify one or more edges of said processed clip. 

19. The method of claim 18, wherein the first value is a 
hexadecimal 00 value, and wherein the second value is a 
hexadecimal IF value. 

20. An apparatus for editing a video program which 
includes a sequence of clips which have been digitized, each 
represented by a frame, the width of each frame representing 
a time duration of the digitized clip represented by that 
frame, and transitions between the clips, the apparatus 
comprising: 

a monitor; 

a mass storage memory for storing the digitized clips; 

a processing means coupled to the monitor and the mass 
storage memory, wherein the processing means is pro- 
grammed with: 

software for displaying a time line, 

software for generating a display representing the video 
program on the monitor, the display including a plu- 
rality of the frames displayed simultaneously on tracks 
in a direct time relationship to the time line, together 
with transition icons representing the transitions 
between the clips, and 

software for editing the video program by modifying the 
display. 

21. The apparatus of claim 20, also including: 

an input device coupled to the processing means, 
and wherein the processing means is programmed with 
software for dynamically generating and displaying 
new frames in response to commands from the input 
device for modifying the display. 

22. The apparatus of claim 21, wherein each of the frames 
representing the clips has a left edge aligned at an in point 
along the time line and a right edge aligned at an out point 
along the time line, and wherein the processing means is 
programmed to change an in point of a first of the clips by 
changing a position along the time line of the left edge of a 
first frame. 

23. The apparatus of claim 21, wherein each of the frames 
has a left edge aligned at an in point along the lime line and 
a right edge aligned at an out point along the time line, and 
wherein the processing means is programmed to change an 
out point of a first of the clips by changing a position along 
the time line of the right edge of a first frame. 

24. The apparatus of claim 21, wherein the display 
includes a first frame displayed in a first track, a second 
frame displayed in a second track with an overlap portion of 
the second frame aligned along the time line with an overlap 
portion of the first frame and a transition icon displayed in 
a third track in alignment with the overlap portion of the 
second frame. 

25. The apparatus of claim 24, wherein the transition icon 
has an associated control icon, and wherein the processing 
means is programmed with software for modifying a param- 
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eter of a transition in the video program between the first clip 
and the second clip in response to manipulation of the 
control icon using the input device. 

26. The apparatus of claim 21, wherein the processing 
5 means is programmed with software for adding to the video 

program a special effect transition between a first digitized 
clip and a second digitized clip, in response to steps of: 
manipulating the input device to display a frame repre- 
senting the first digitized clip in a first track, and a 
io second frame representing the second digitized clip in 
a second track with an overlap portion of the second 
frame aligned along the time line with an overlap 
portion of the first frame; 
manipulating the input device to select a first transition 
15 icon; and 

manipulating the input device to position the first transi- 
tion icon in a third track in alignment with the overlap 
p portion of the second frame. 

27. The apparatus of claim 26, wherein the processing 
20 means is programmed with: 

software for displaying a menu of transition icons includ- 
ing the first transition icon; and 
software for selecting the first transition icon in response 
25 to manipulation of the input device. 

28. The apparatus of claim 27, wherein each of the 
transition icons of the menu is an animated icon which 
simulates an available transition in the video program. 

29. The apparatus of claim 21, wherein the processing 
30 means is programmed with: 

software for displaying a filter menu having control icons 
representing parameters of a temporally varying filter 
for filtering a selected digitized clip; and 

software for determining the parameters of the temporally 
35 varying filter in response to manipulation of the control 
icons using the input device. 

30. The apparatus of claim 29, wherein the processing 
means is programmed with software for filtering the selected 
digitized clip with the temporally varying filter. 

40 31. The apparatus of claim 29, wherein the temporally 
varying filter is a mosaic filter, and the parameters include a 
start level and a final level for the mosaic filter. 

32. The apparatus of claim 20, wherein a first one of the 
transition icons is displayed in a special effects track, and 

45 has a left edge aligned at an in point along the time line and 
a right edge aligned at an out point along the time line, and 
wherein the processing means is programmed to change the 
in point of the first one of the transition icons by changing 
a position along the time line of the left edge of said first one 

50 of the transition icons. 

33. The apparatus of claim 20, wherein a first one of the 
transition icons is displayed in a special effects track, and 
has a left edge aligned at an in point along the time line and 
a right edge aligned at an out point along the time line, and 

55 wherein the processing means is programmed to change the 
out point of the first one of the transition icons by changing 
a position along the time line of the right edge of said first 
one of the transition icons. 

34. The apparatus of claim 20, also including: 

60 a cache memory coupled to the processing means, and 
wherein the processing means is programmed with 
software for identifying data needed to modify the 
display in response to each of the commands for 
modifying the display, then retrieving from the cache 

65 memory at least a first portion of the data currently 
stored in the cache memory, then retrieving any remain- 
ing portion of the data from the mass storage memory, 
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then processing the retrieved data to generate processed 
data for modifying the display, and caching the 
retrieved data in the cache memory. 
35. The apparatus of claim 20, wherein the processing 
means is programmed with software for performing special 5 
effects processing operation on a first digitized clip and a 
second digitized clip the operation including the steps of: 
filling an alpha* channel of the first digitized clip with a 
first value; 

filling an alpha channel of a second digitized clip with a 10 
second value; 



performing a first special effect process in which selected 
data of the first digitized clip are combined with 
selected data of the second digitized clip to generate a 
processed clip having an alpha channel; and 
processing data in the alpha channel of the processed clip 

to identify one or more edges of the processed clip. 
36. The apparatus of claim 35, wherein the first value is 
a hexadecimal 00 value, and wherein the second value is a 
hexadecimal IF value. 
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(57) ABSTRACT 

A non-linear editing (NLE) system for editing and/or modi- 
fying 3D animation information comprises elements repre- 
sented by clip objects which can be positioned and/or 
manipulated relative to a time line. The elements can com- 
prise conventional ID (audio) or 2D (video) information or 
can comprise 3D animation information which can include 
animation objects and animation parameters associated with 
them, as well as 2D renderings of those objects. Positioning 
and/or manipulation of clip objects representing 3D anima- 
tion elements can result in alteration of the 2D rendering and 
or re-rendering of the 3D information, as appropriate, 
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METHOD AND SYSTEM FOR EDITING OR 
MODIFYING 3D ANIMATIONS IN A 
NON-LINEAR EDITING ENVIRONMENT 



FIELD OF THE INVENTION 

The present invention relates to a system and method for 
editing or modifying 3D animations. In particular, the 
present invention relates to a non-linear editing system and 
method for editing and/or modifying 3D animations. 

BACKGROUND OF THE INVENTION 

Non-linear editing (NLE) systems are known for video 
(2D) and audio (sometimes referred to as "ID") systems. 
NLE systems for video and/or audio include systems manu- 
factured by AVID, Quantel and others. Non-linear video 
editing systems allow a user to join, arrange and/or modify 
digital or digitized source information to create a final cut, 
or edit, for rendering to appropriate storage media or output. 
The systems are non-linear in that the sources are not 
generally limited to being accessed in a linear manner, but 
instead permit random access. For example, the last three 
seconds of a ten second digital video source can be accessed 
without accessing the first seven seconds, or individual 
frames in the digital video can be accessed and/or combined 
as desired. Non-linear audio editing systems allow a user to 
easily and efficiently manipulate digital or digitized audio 
sources to produce a desired final output, often referred to as 
a final edit or "cut". 

In practice, an NLE system for video can, for example, be 
used to construct the video portion of a television adver- 
tisement for a new model of car, by connecting digitized 
source video of the new car driving on a country road, digital 
source video of a driver in the car and digitized source video 
of the car in a wind test tunnel with engineers standing about 
it. An NLE system for audio can be used to create an audio 
soundtrack for this advertisement, comprising a selection of 
classical music, in digital form, mixed with a narrator's 
voice-over, also in digital form, discussing the features of 
the car and various sound effects sources which are also 
mixed, at appropriate time points, with the music and 
narration sources. The video and audio final edits are then 
rendered to an appropriate sink, such as a VTR, RAID array, 
or a monitor, to obtain the finished commercial. 
/ NLE editors represent the available sources (video or 
Uaudio) as objects in a graphical user interface (GUI), the 
Jobjects commonly being referred to as clips, which are 
' positioned and/or manipulated with respect to a time line for 
the edit. Various effects, such as fades, dissolves, wipes, 
fiends, etc. can be applied to the clips to obtain the final edit. 

Known techniques for editing 3D animations have com- 
prised the rendering of the 3D animation to 2D media 
(video) which is then edited in a conventional manner, such 
as with the above -described NLE. The 2D media rendered 
from the 3D animation is treated as any other 2D source 
material, such as digital video, and can be combined with 
such other source material, as desired by the director. 

If the director requires a change to the 2D rendering of the 
3D animation, for example to have a character walk past a 
fixed background at a faster rate, or to have the contents of 
the 3D animation composited together in a different order, 
the director must instruct an animation artist as to what is 
desired and the animation artist will then invoke the required 
animation tools lo modify the animation and to re-render it 
to 2D media. This new 2D material would then be provided 
to the director to replace the previous material. 
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While these techniques have provided reasonable results 
in the past, it is desired to have a method and system of 
editing and/or modifying 3D animations which is easier to 
employ, more efficient than prior art editing techniques and 
which provides a flexible creative environment for directors, 
editors and artists. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide a novel 
system and method of editing and modifying 3D animations. 

The present invention provides a method and system for 
editing and modifying animation information which reduces 
or removes the barriers between a director and the source 
material he is working with. Conventionally, with 2D video, 
which can include 2D renderings of 3D animations, NLE 
systems are used to edit existing source information. 
Essentially, the director creating the final edit is only able to 
select, arrange and apply effects to source information which 
has already been created. If additional source information is 
required, the director must undertake to have such additional 
information created, by fil mijug new scene s, etc. and there is 
thus a barrier between the director and his work. The present 
inventors have determined thai an Nl!ti wnicn is used to edit 
source material from 3D animations need not be so limited 
and that the barrier experienced with such conventional 
systems can be mitigated. Specifically, the present inventors 
have created a system and method for the editing and 
modification of 3D animations which allows an NLE system 
to integrate the 2D representations which the director is 
dealing with to the 3D world of the animations. As described 
below, with the present invention the director or other user 
of the NLE system can edit and manipulate both 2D ren- 
derings of 3D animations in a conventional manner and can 
also edit, manipulate and re-render those 3D animations as 
necessary. 

According to a first aspect of the present invention, there 
is provided a computer implemented method for creating 
and modifying an edit comprising at least 3D animation 
information, comprising the steps of the user selecting an 
element from a set of available elements, the element 
representing 3D animation information which includes at 
least one parameter, such as a start position, end position, • 
animation speed, etc. The selected element is represented to 
th e user as a clip object which the user can p os itioja^in. a 
gra phjcj^l^d i srjlay ot trie r computer ^lati ^to a ^ime|Un e. The 
size of the clip object" in the graphical cire^ to the 

ti me line corresponds to a duration whirh is applififl tn 
a ppropriate 6nes.Q.tJb,e ^aram eJers inthe element, aqd jhe^ 
e dges of the clip object relative to the time r line_represejiL a. 
star t ti m e jind a n en d ji m e^e ach„o LwJlich_are J a lso_a ppJ i ed^ to 
appropriate ones of the parameters of the element. The 
computer accesses the animation information, in accordance 
with the information applied to the parameters, to create the 
edit by compositing together each element. Modifications \o\ 
the edit can be achie ved by the user repositioning and/or 1 
resizi ng the clip objects^ the^co.mpuier ap plying the , corre- 
sponclirig changes t o the a^pjroprjate^^ 1 
re-compositing the edit, and by removing, adding or substi- 
tuting elements which will be composited into the edit. 

According to another aspect of the present invention, 
there is provided a computer implemented non-linear editing 
system for creating and modifying edits comprising at least 
3D animations, comprising: a storage device to store ele- 
ments including animation information and parameters 
which relate to the animation information; a computer 
operatively connected to the storage device to access the 
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elements; one or more output devices to display to a user a interface (GUI), described further below, of the NLE system 

graphical user interface of an NLE, including a time line, and the outputs of the edit being produced are each 

and to display the result of an edit produced with one or displayed, either simultaneously or alternately, on monitor 

more of the stored elements which are composited together 16 with any associated sound track being output by sound 

to form the edit. The system also includes at least one user 5 system 18. It is also contemplated however, that in some 

input device with which the user can: select at least one circumstances more than one monitor 16 may be provided to 

element from a list of said elements stored on said storage. allow 0Ut P ut 10 be displayed on one or more monitors while 

device; define the positioning, relative to the time line, of a the GUI of the ^ svstem is displayed on another, 

clip object representing the selected element, the start time, FIG - 2 15 a block diagram of the data relationships 20 of 

end tim e and duration of the animation producedJrpjrU he 10 a non-linear editing (NLE) system, in accordance with the 

element being determined,byJhe a positionmg anTshSg of P resenl invention. In the following discussion, an edit cre- 

the clip object reTSuyeaolthe^Ome^Jine. The computer ated b y NLE svstem 10 1S referred to as a project 24. A ; 

accesses each element in correspondence with the position- P ro J ecl 24 comprises a series of elements 28, such as video, 

ing and sizing of its respective clip object, and composites audl0 > sllU ima S e ' stalic 30 ^formation and/or 3D anima- 

the elements together to produce the edit. .Wherua._user 15 tions wnich can be combined within project 24 and then 
re positions and/or resizes one on more r clip obiects^he^ rendered, mixed and/or composited to produce the final edit, 

chan ges ULlhe startjjmjG^ In the ^ re > ^ree elements 28 are shown: first element 2Sa 

elemen^orrespS madelo thecHps are can be audl ° information, second element 286 can be 

ap plied the appropriate oaramel gre aTrT'eTe ^^^ the information for a still image and third element 28c can be 

computer re^cjgsses^ 20 ^formation for a three-dimensional animated character. As 

according to the new values ofl^aTa^eTe^: * indicated in the Figure, project 24 can include more than, 

*&**: W z<*z n **** x T rr* three elements 28 and can include, for example, multiple 

1 BRIEF DESCRIPTION OF THE DRAWINGS elements of audio, video and/or 3D animation, still images 

and static 3D objects. Further, as will be described below, 

Preferred embodiments of the present invention will now dements 28 of project 24 can also themselves be projects.' 

be described, by way of example only, with reference to the Vmh ^ each elemem 28 can also have one or more defined 

attached Figures, in which: associations with appropriate tools 32, as discussed in more 

FIG. 1 shows a representation of an NLE system in detail below. In addition to elements 28, project 24 includes 

accordance with the present invention; an edit data structure 30 which maintains the 

FIG. 2 shows a block diagram of a data structure of the 30 interrelationships, defined by the user with system 10, 

NLE system of FIG. 1; between elements 28 to produce the final edit and which 

FIG. 3 shows a graphical user interface of the system of maintains the associations between elements 28 and the 

PIG j. appropriate tools 32. 

HG.' 4 shows an enlarged schematic view of the time line . r Pr0 -) ect 24 , caD include « lements Mwbich utilise source 

area of FIG 3- 35 u " ormat,on " om storage devices, such as RAID Array 14, 

„_ , , digital video or audio tapes, etc., and project 24 can itself be 

FIG. 5 .s a block diagram of an animation element in stQred Qn , devices such as ^ ^ 14> e 

accordance w.th the present mvention; and devices connected to mlnpulef syslem n by n6twork ^ 

FIG. 6 is a block diagram of a project data structure. nection 15 etc. 

DETAILED DESCRIPTION OF THE 40 Tools 32 can include an animation program, such as the 

INVENTION SoftImage|3D product sold by the assignee of the present 

invention, or components thereof, compositing systems, one 

FIG. 1 shows an example of an NLE system 10 in or more rendering engines, such as the "mental ray" ren- 

accordance with an embodiment of the present invention. dering engine included with the above-mentioned 

System 10 includes a computer which, for example, can be 45 SoftImage|3D product, etc. As is known by those of skill in 

a general purpose computer system 11 such as a PC with an the art, the choice of rendering engine depends on the 

Intel Pentium processor and executing Microsoft Windows desired quality, speed, resolution and other related factors 

NT operating system, or a special purpose computer system, and those of skill in the art will understand the similar 

and which execute a series of program steps to provide an considerations to be made in selecting other tools 32. As 

NLE-based system for editing and modifying 3D anima- 50 mentioned above, each element 28 can have an association 

tions. Computer system 11 is connected to one or more user with one or more appropriate tools 32. For example, an 

input devices, such as keyboard 12, mouse 13, or any other element 28 which includes animation information for a 3D 

suitable user input device such as a graphics tablet (not animated character can have a defined association to the 

shown), etc. While computer system 11 will generally animation component of the above-mentioned 

include a non-volatile storage device, additional data storage 55 SoftImage|3D product which allows system 10 to invoke 

for storing source information, edits in progress and final tool 32, as necessary, to modify the animation information, 

edits can be provided by a storage device such as RAID These Associations can be defined in a variety of manners as 

array 14, optical disc drives (not shown), digital or analog will occur to those of skill in the art and can, for example, 

video or audio tape recorders, etc. be defined according to extensions to filenames storing the 

As will be apparent to those of skill in the art, computer 60 information, defined according to storage locations, i.e. — a 

system 11 can be directly connected to storage devices, such subdirectory can be provided for animations, a different 

as RAID array 14, and/or be connected to storage devices subdirectory can be provided for still images, defined explic- 

via a suitable communications link, such as LAN network itly for the information by the user, etc. 

connection 15, via the internet, etc. System 10 also includes A user interacts with NLE system 10 through an NLE 

one or more output devices, such as high resolution monitor 65 graphical user interface (GUI) 40, an embodiment of whicli^ 

16, sound system 18, etc. and their related I/O cards (not is shown in FIG. 3. As shown, GUI 40 consists of an imagingy 

shown). In the embodiment of FIG. 1, the graphical user area 44, function area 48 and an NLE time line area 52.1 
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Imaging area 44 can be a portion of a monitor screen or it 
can be a separate monitor, such as a high resolution monitor 
and imaging area 44 can display the contents of visual 
(video, still image, animations, etc.) clips, or portions of 
those clips, the visual output of the complete current edit, 
etc. Similarly, sound system 18 can output the contents of 
audio (music, narration, sound effects, etc.) clips, or portions 
of those clips, the audio output of the complete current edit, 
/etc. In this manner, a director can review the contents of 
J clips, crop or otherwise edit them as appropriate (described 
<below) and view the results. 

V , Function area 48 consists of, for example, function but- 
tons 60 and function menus 64 associated with a particular 
tool 32, if such a tool is active, or with system 10. It is 
contemplated that, in many circumstances wherein a tool 32 
is accessed to modify a clip, via the association defined 
between the clip and the tool 32, function area 48 will be 
updated to display at least a subset of the commonly used 
functions and operators of that tool 32. For example, in an 
animation application, the functions can include shading, 
scaling, predefined movements, and other effects, as well as 
general file retrieval and the like. 

If no specific tool 32 has been invoked, function area 48 
displays functions and operators relating to system 10 and 
this can include an Explorer-style browser, as shown in the 
Figure, which allows a user to select projects of interest, 
. such as the illustrated Scenes 7 through 12, or available clips 
^ to be operated on with system 10. 

time line area 52 is shown schematical ly., in FIG. 4^ 
and thumbnails 84 have been orn)']tfifiiifor.rJar.it)wNTiB^irnf > 
line area 52 displays a jjroject 24 for "Scene 8" o f FIG. 3 in 
graphical form in relalliHl 115 U luiie line 4b where time 
adv ances from le ft tojjjfih. t . Elements 28 are displayed as 
ieparate rectangular clips 68, each in a track 72. As 
described above for elements 28, clips 68a, 686 and 68c can 
comprise audio information, video information, still image 
information (such as a background scene) or static 3D 
information (such as a 3D set through which an animated 
character walks) and animated 3D object information, such 
as an animated character or other model, respectively. In the 
specific time line for the project "Scene 8" shown in FIGS. 

V3 and 4, clip 68a represents an element 28a of a still image 
background, clip 68b represents an element 28b of an 
animated 3D character, in this case a man, and clip 68c 
represents an element 28c of a light source. 
*^In this specific example, element 28a can have an asso- 
ciation defined with a paint tool 32, to allow an editor to 
modify the still image, and to a rendering engine tool 32 so 
that element 28a can be included by the rendering engine 
tool, should it become necessary to re -render the project as 50 
a result of changes to elements 28 made within system 10. 

Each clip 68, and the underlying element 28 that it is 
associated with, have certain features in common. Each clip 
68 is represen ted by a two-dimensional rectangular box , and 
each has, in retation to time line 66, a start point _7iL and an 55 
end point 80 defining a duration therebe tw een. The start 
point 76, end point 80 and duration of each" cl ip 68 is mapped 
to appropriate points in the correspondina element 2 8 which 
caT^eJa rfte r n mj^ 

its c orresp_QndinR^ljpJ jj8. E ach track 72 can have more than 60 
one clip 68, or more th an o n e insta nce o f Jhe same clipj as 
shown in tfieTigur es > wh^eln' t wo ^ 

track. When two instances o^'a clip are included, they can be 
either repeTnTStffoveTia^ing or different portions of the total 
duration of the element 28 associated with the clip 68. 65 

While the concepts of NLE-type editors are known for 
video and audio, the present invention employs the NLE 
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metaphor to edit and modify 3D animation information, 
effectively providing a convenient and efficient 2D link to 
3D information. 

As shown in FIG. 3, in a presently preferred aspect of the 
invention each clip 68 can also have associated with it 
thumbnails 84 which are preferably graphic representations, 
or icons, representing the content of the clip, although they 
can also be a textual description or both. When graphical 
representations or icons are employed, these thumbnails 
84a, 84b, 84c can indicate the change or evolution in the 
underlying information over time, if any. For example, the 
thumbnail 84a, which shows an icon of a background, 
represents the information in element 28a. The information 
in clip 28a does not change over time, therefore, a single 
repeating thumbnail is sufficient to represent the informa- 
tion. By contrast, thumbnails 84b on clip 68b, are changing 
icons representing the state of the information in element 
28b, which is associated with a 3D animation of the char- 
acter "John" who walks as the animation proceeds. Clip 68c 
represents an element 28c which, in this example, comprises 
information relating to a light source to be included in the 
rendering of the project and, in this specific example, 
thumbnail 84c occurs twice along time line 66 so that the 
light source has two separate durations. This has the effect 
that the light is "on", i.e. — included in any rendering of the 
animation, for a first duration within the edit and "off", 
i.e. — not included in the rendering of the animation, for a 
subsequent duration within the edit and then "on" again for 
another duration of the edit. 

To produce an edit and/or to modify the information in"an\ 
element 28, a user modifies or operates upon the correspond- 
ing clips 68 in the NLE time line area 52. A user can select 
elements to be included in an edit from a list of available 
elements presented in the browser of function area 48. The 
user can drag a desired element from this list and drop it onto 
a track 72 where it will be represented by a clip 68. A user 
can change the duration of a clip 68 by scaling clip 68 to the 
desired length. This can be accomplished in a wide variety 
of manners, as will be understood by those of skill in the art, 
including well known "dragging" operations wherein a user 
clicks on either the start or ending edge of clip 68 with an 
input device such as mouse 13 and drags the edge to the 
desired new position. Similarly, a clip 68 can be moved 
along its associated track 72 to change its start point 76a, 
16b, 16c and end point 80a, 80/?, 80c with respect to time 
line 66 using keystroke combinations, mouse manipulation, 
or by any other suitable means as will occur to those of skill 
in the art. If a clip is shortened, the duration of the source! 
information in corresponding element 28a which is used in 
the project is reduced. If a clip 68 is shifted to the left, wit 
respect to time line 66, the source information in corre 
sponding element 28 is used earlier within the project th 
before. 

Unlike clips representing 2D or ID information, clip 68b 
represents 3D animation information, and can be manipu- 
lated via NLE system 10 in an enhanced manner, either as 
a 2D rendering of the 3D animation, if appropriate, or as 3D 
information which can be manipulated and subsequently 
re-rendered, if appropriate. For example, clip 68b can be 
lengthened to increase its duration, as with clips for 2D or 
ID information, but this can result in element 28b being 
re-rendered from 3D information to a 2D representation. 
Specifically, if element 286 represents an animated 3D 
model of a bipedal character, such as "John", walking from 
left to right, lengthening clip 68b can result in a decrease of 
the speed at which the character walks, or an increase in the 
distance travelled, depending on animation parameters 
included with the animation information in element 28b. 
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The term "animation parameters", as used herein, is It is further contemplated that animation elements 28 can 

intended to comprise the various components which define have default conditions defined by the animator for modi- 

an animation element. Animation parameters can be either fication of duration or for other modifications. For example, 

fixed, i.e. — not being directly changeable via the NLE, or in the above-mentioned example of a character walking 

modifiable, i.e. — able to be modified via the NLE. For 5 between a start and end position, the default condition can 

example, FIG. 5 depicts an animation element 88 for an be to extend beyond the start and end positions, i.e. — to start 

animated figure walking in a left to right direction. Anima- walking from a position before the start position and/or to 

tion element 88 includes a skeleton parameter 92 comprising walk toward a position beyond the end position. Other 

the definition of an articulated chain hierarchy and a flexible appropriate default conditions can be defined for animations, 

envelope, or skin, parameter 96 which is applied to skeleton as will occur to those of skill in the art. In this manner, 

parameter 92. Both the skeleton parameter 92 and flexible duration or other modifications can result in modifications to 

envelope parameter 96 are fixed. the animation in a manner defined by the animator as being 

When fixed parameters require modification by an appropriate for the particular animation. It is contemplated 

animator, the NLE does allow the animator to invoke the that the definition of default conditions can be performed, in 

required tool 32, as specified by the associations for element any suitable manner including the completion of a default 

88, as needed. For example, if an animator wishes to modify condition dialog box by an animator as an explicit step when 

the articulated chain hierarchy of skeleton parameter 92, a the animation information is defined, or when the animation 

modelling tool 32 can be invoked from the NLE time line, mformation * first used in system 10. 

via a suitable user input event such as double-clicking on the ~, . i. L • 1 1. • j- . . . . LJ 

clip, and the skeleton modified as desired. In such a case, ^ aclual tec °mques employed to indicate which method 

Function area 48 can be updated to display at least a subset 20 ° f increasing or decreasing the duration of element 88 is 

of the commonly used functions and operators of the mod- desired ue not Particularly limited. For example, selecting 

elling tool 32 lne C ^P representing element 88 with a left mouse button can 

A fixed repeated walk cycle parameter 100 can be defined correspond to maintaining the start and end position fixed 

for a repeatable sequence of skeletal articulations necessary and changing the speed while selecting the same clip with 

to produce a continuous walking motion for the animated 25 the ri S ht raousc butt0D can correspond to maintaining the 

figure represented by element 88. In addition, a walking speed constant and modifying the start or end position. Other 

speed parameter 104, a start position parameter 108 and an suitable methods for specifying the mapping of changes 

end position parameter 112 can be defined and set as a clip to parameters in the corresponding element will 

modifiable parameters associated with the fixed parameters. apparent to those of skill in the art. 

In some cases, animation parameters are inherently set as 30 A modified animation clip may have to be re -rendered to 

being of fixed or modifiable type. For example, the skeleton, provide a sequence of animation frames in accordance with 

flexible envelope and similar parameters can be inherently the desired modification to its animation parameters. In th4 

set as being fixed types as these parameters will not be examples above, where the duration of a clip is increased oi 

modified by a director/user in the normal course of creating decreased, the number and/or content of the frames in thel 

an edit. Similarly, parameters relating to speed, position and 35 modified element , will change and must be re-rendered^ 

other like factors can inherently be set as being of modifiable unless the modification was limited to the discarding of a 

types as these parameters will commonly be modified by a portion of the animation or the employing of additional 

director/user in the normal course of creating an edit. In any frames, previously rendered and stored. Re_- rendering. 

event, each parameter will have at least one appropriate tool a nimations in response to modifications can be performed 

32 associated with it to permit modification of the parameter. 4 q real ti me or can be deferred until an appropriate selected 

Further, each parameter can explicitly be set by a user as time,~TOr example alter multiple modifications have been 

being fixed or modifiable, if such should be desired, by any periormed and it is desired to view the results. Re-rendering 

appropriate user input. can be performed by one or more rendering engine tools 32 

In the NLE of the present invention, element 88 is associated with the element requiring re-rendering, 

represented as a clip, such as clip 686, which can be 45 As mentioned above, in addition to modifying the length 

modified in a variety of manners. For example, if the clip for or position of a clip, the fixed animation parameters of an 

element 88 is lengthened this indicates that the duration of element can be modified by opening the element, such as by 

the animation is to be increased. However, the animation double-clicking on it with a mouse, to invoke an associated 

corresponding to element 88 can be lengthened in at least tool 32 to access and/or modify the fixed parameters, 

two manners. For example, speed parameter 104 can be 50 Further, new fixed or modifiable parameters can be added to 

decreased and start position parameter 108 and end position a clip by, for example, dragging and dropping functions from 

parameter 112 held constant, with the result that the char- function area 48 onto a clip. For example, various — 

acter in the clip walks the distance between the start and end functions can be provided to modify the result 

position with a slower gait, i.e. — the animated character animation. A JITTER function can be applied to an 

covers the same distance within the increased duration of the 55 tion to add random noise to the motions therein to 

clip. Alternatively, speed parameter 104 can be held constant the rendered animation or a CHARCOAL function can be 

and the position of either start position 108 or end position applied to the animation to give the rendered animation the 

112, corresponding to which end of the clip is altered, is appearance of having been drawn as a charcoal sketch, 

moved to increase the distance travelled by the animated Effects or functions can be applied to any clip or to an entire 

character within the increased duration of the clip. project and can thus affect part of the final edit produced 

If the duration of a clip is shortened, corresponding results with the present invention or the entire final edit, 

can be obtained whereby speed parameter 104 is increased, In addition to modifying the parameters of any given 

so that the animated character walks the same distance element in a project, a user may also m odify the relationsh ip 

between the start and end positions within the shortened betweenelements in a project, and add, su btract or substitute 

duration, or the start or end positions can be modified so that 65 ele ments' within a 1 pMltflH. In" pflrttEfl&r: an edit" can be 

the animated character walks less distance in the shortened periormed with low quality elements, for speed and perfor- 

duration. mance considerations, or elements which are merely place - 
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holders for information which is not yet available. Once the musicians (i.e. — three guitarists and a drummer); an element 

information becomes available or a final edit is required at representing a background (i.e. — a still image of a sky with 

a different quality level, the various elements in the project clouds in it); and an element representing an animated 3D 

can be replaced by the desired elements without requiring blimp flying over the stage. 

any other effort on the part of the animator. In such a case, 5 In the NLE system in accordance with the present 

i he user can select one or more desired elements from a list invention, each of these elements can have a track associated 

i )f availab le elements present e d in the browser in Func tion with it and the elements are displayed as clips relative to the 

irea 48 aria drag and drop tne^es'irea' element on top of the time line for the edit. While the static elements of the 

< lip representing the placeholder or different quality element animation, such as the stage, are not manipulated to any 

i l a track 72 in NLE time line area 52. When a desired clip in great extent within the NLE system, the animated elements 

ii dropped onto a clip already in a track ?1 the desireajcup^ can be manipulated as desired. For example, the animators 
repl a ce s ^ and the sta r t time^^nd can have defined the 3D animations of the musicians such 

1? a^dural'ion ot "Ine "de sired'cii p are "seTto t hose o L the that the drummer walks onto the stage from the left side and 

"jg^g^soV"any*ertects^ wnich sits down at his drunks and begins to drum. A first guitarist 

tto the clip previously in place are automatically J5 runs onto the stage from the right side, picks up his guitar 

1 to the desired clip. and starts to play. A second guitarist leaps onto the stage 

' Further, referring to FIGS. 3 and 4, in a present embodi- from the left side, runs to his guitar, picks it up and begins 

ment of the invention the relationship, or layering, of to play. The third guitarist runs onto the stage from the right 

elements in a project is determined by the order in which side, stops at the center of the stage and bows to the audience 

tracks 72 are displayed within NLE 40. The elements are 2 q ana< tnen turns 10 P* c k U P bis guitar and begins to play. Each 

composited together in the order in which they are displayed musician is defined as an animated bipedal model by the 

from top to bottom, the uppermost element thus forming a animation artists. 

first composited layer with successive tracks being compos- In this example, using the NLE system in accordance with 

ited in respective order as successive layers. Thus, the present invention, the director can change when the 

re-ordering the tracks 72 will result in a comparable change 25 various . events occur, i.e. — how long after the drummer 

to the order in which elements are composited together. It is starts to play that the first guitarist walks onto the stage, by 

contemplated that any other suitable means of ordering the positioning or repositioning the start of the respective clip 

layering of elements, such as explicitly assigning a layer to elements at different points on the time line. The director can 

each element, can also be employed if desired and, in such change the duration of events, i.e. — how long it requires the 

cases, the ordering of tracks 72 in NLE time line area 52 is 30 drummer to walk across the stage to his drums or how long 

independent of the compositing order. the third guitarist bows to the audience, by lengthening or 

Subtracting an element from a project is simply accom- shortening the clips to speed up or slow down the animated 

plished by removing it from its track and optionally, if it is movement. The director can also re -arrange the compositing 

the only element on its track, removing its track. Adding an of the musician models on the stage, moving the drummer 

element involves dropping an element from a list of avail- 35 from the back of the stage to the front of the stage, etc, by 

able elements or from another track onto an existing track re -ordering the tracks in the NLE system. The director can 

and/or adding a new track to receive the element, as appro- also select various cameras from which to view the 

priate. Similarly, substituting an element can be accom- animation, i.e. — the viewpoint can switch between a camera 

plished by dropping a new element onto an existing element which is constrained to the moving blimp object, and which 

in a track, as described above. 40 provides an aerial view of the stage, and a camera showing 

A special instance of an element in the present invention a front view of the stage. The director can also turn light 

is a camera. Essentially, a camera is a viewpoint from which sources on or off or otherwise modify them, 

a scene is to be rendered. Multiple cameras can be defined Unlike prior art NLE systems, in the examples above the 

for a 3D animation, to allow a director to cut between director can actually manipulate the 3D information, via 2D 

different viewpoints, and cameras can have positions which 45 information displayed in the NLE system, to obtain the 

change, i.e. — the camera follows a defined path, or is desired final 2D rendering of the animation. As will be 

constrained to move with another object, i.e. — the camera apparent to those of skill in the art, some manipulations of 

can be mounted on the shoulder of an animated character. the 2D information displayed in the NLE system will require 

Camera elements are represented as clips in tracks, like other re-rendering of portions of the animation. For example, 

elements in the NLE system of the present invention, but are 50 slowing the drummer's walk to his drum set will require 

principally manipulated by selecting between the available re-rendering that portion of the animation of the drummer to 

cameras. In some instances, a camera can follow an ani- add the additional frames required. Ho wever, as will a lso be 

mated path and manipulation of the clip representing such a apparent, this will not re quire re-renc| ering | omna r^cfed 

camera element can be used to alter the camera's speed p ortions of the total animation, i .e. — changing the speed at 

along the path. Like other elements in the present invention, 55 which the drummer walks does not require re-rendering of 

if the director should wish to change a camera element in a the guitarists, although it may require repositioning their 

manner requiring the use of an animation tool, such as clips with respect to the time line. As win jx apparent to \ 

changing the path followed by a camera or changing the those of skill in the art, selecting a duraiidn^oVaiL element I 

object to which it is constrained, the associated tool can be wh ich is less than the entire duratiQjL_alteMv^endexed I 

invoked from within the NLE system. 60 (i.e.— selecting only five seconds of an _ eig^ fi fipond I 

The present inventors believe that one of the particular animation ) win not re quire any re-rendering^ nor will sub^ 

advantages of the present invention is the use of an NLE to sequenUv.jerselecung.s ome of the unused "re TnBejftd^r^U'Qa l. [ 

manipulate 3D information from a 2D representation Other manipulations can require re-rendering, to a different 

thereof. For example, if a 3D animation of a rock music viewpoint for example, or re -compositing of the rendered 

concert is to be produced, the animation artists can have 65 elements of the animation. 

created an element representing the stage (i.e. — a static 3D When re-rendering is required, it is contemplated that a 

object); elements representing each of the animated 3D variety of techniques are available to deal with this in a 
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user-friendly manner. For example, a limited re-rendering 
can first be performed to produce updated thumbnails on the 
clips in the NLE system, to provide quick feedback to the 
director, and a subsequent complete re-rendering of the 
changes can be performed in the background, either auto- 
matically or when explicitly initiated. It is contemplated that 
this latter case (full re-rendering only when explicitly 
initiated) may be desired when the director is experimenting 
with a final cut and a full rendering is not required until a cut 
of interest has been produced. Further, wire frame or low 
quality re-renderings can be performed to provide quick 
feedback to the director and full re -renderings of the desired 
quality can be subsequently performed. When the director 
requires a change which requires an animation tool, such as 
for example changing the path that the blimp flies over the 
stage, the appropriate tool will be opened by the NLE system 
to allow the animator or the director to effect the desired 
changes. 

It will be apparent to those of skill in the art that the 
present invention offers a director or other user a familiar 
interface for creating a desired final edit, yet the operations 
performed by the interface can directly map to 3D animation 
information, rather than to 2D renderings thereof. It is 
believed that this provides significant advantages over con- 
ventional NLE systems which merely manipulate 2D or ID 
information. 

FIG. 6 shows a representation of a data structure 200 for 
a project. As shown, each project data structure 200 includes 
a project description 204, in this example "Rock Concert", 
a pointer 208 to a set of project properties, data 212 
representing the number of elements in the project and a 
pointer 216 to those elements 220a, 2206, . . . 220/t 

The project properties contain information relevant to the 
project as a whole and can include total project duration 
information 224. This duration information is represented in 
seconds, microseconds, as a number of frames, etc., as 
appropriate. The location 228 of the rendered information 
(video, animation, audio, etc.) and any other relevant infor- 
mation is also present in these properties. 

Each element 220a, 2206, . . . 220/i points to an appro- 
priate element data structure 232. Each data structure 232 
includes a type identifier 236 which identifies the type of 
element, i.e. — audio data, video data, still image, 3D ani- 
mated object, etc., and an element identifier 240 which 
uniquely identifies that element, i.e. — "lead guitarist", 
"drummer", the title of the audio track "jump", etc. 

Each element also includes an offset 244 which indicates 
the start of the element relative to the start of the project. For 
3D animated objects, this offset is indicated as a number of 
frames, as is the duration information 248, while for audio 
information it is indicated as a time or as a number of 
samples, as appropriate. Location information 252 indicates 
where the relevant information is located, including the 
model definition and animation parameters for a 3D object, 
the storage location for audio or video information, etc. 
Additional information 256, such as a preferred rendering 
engine and other associations, can also be maintained as 
appropriate. 

As mentioned above, it is also contemplated that projects 
can themselves be elements of a project. That is, a first 
project can be created and edited as above and then inserted 
as an element in a larger project. In the larger project, the 
first project will be represented as a clip which can be 
expanded to show the corresponding parameters in an NLE 
and modified as described above or which can be manipu- 
lated as a single clip within the larger product in a similar 
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manner to any other clip. For example, in the above- 
mentioned Rock Concert animation example, the third gui- 
tarist's walking on to the stage can comprise one element, 
his bow to the audience a second element and his playing his 

5 guitar a third element. These three elements can be com- 
bined into a single project which is then incorporated into 
the Rock Concert project. If it is desired to modify or 
otherwise manipulate the individual elements of the third 
guitarist's actions, the third guitarist project is opened to 

10 access the tracks of the individual elements. Once the 
desired modifications have been effected, the tracks of the 
third guitarist's actions can be collapsed again to a project 
which is represented as a single clip in the overall project of 
the Rock Concert. This single clip can also be operated on 

is to alter the start time of the third guitarist project (i.e. — to 
change when he walks onto the stage), or otherwise manipu- 
lated as desired. 

Though this discussion has focused on modifying 3D 
animation information, it will be appreciated that the present 

20 invention can apply to any other type of information which 
can be defined as an element having parameters. As each 
element in a project is independent and can be modified and, 
if necessary, re -rendered independently of other elements in 
the project, a variety of elements from different information 

25 sources can be edited simultaneously within a single con- 
sistent user interface. It will also be apparent that the present 
invention can be employed in combination with 2D or ID 
information as desired. For example, an audio soundtrack 
can be edited for the rock concert example given above 

30 and/or video (such as actual filmed crowd scenes) can be 
included in the compositing of the final cut. 

It will be apparent to those skilled in the art that the 
foregoing is by way of example only. Modifications, varia- 
tions and alterations may be made to the described embodi- 

35 ments without departing from the scope of the invention 
which is defined solely in the claims. 
We claim: 

1. A method for creating and modifying an edit compris- 
ing 3D animation information, comprising: 

40 providing a plurality of animation elements with associ- 
ated animation parameters, wherein the animation 
parameters include the 3D animation information; 
selecting an animation element from the plurality of 
animation elements; 

45 

generating a clip object by arranging the selected anima- 
tion element in an associated track for graphical display 
relative to a timeline; 
compositing a plurality of the clip objects so as to form a 

50 collection of layered tracks defining the edit, wherein 
one of the clip objects includes a sequence of 2D 
animation frames associated with the 3D animation 
information and having a start position and an end 
position relative to the time line; 

55 modifying at least one of the animation parameters asso- 
ciated with the selected animation elements and inde- 
pendently re-rendering the associated animation ele- 
ments in accordance with an extent of each 
modification, wherein modifying includes modifying a 

60 duration of one of the clip objects by altering one of the 
associated start and end positions of the clip object, and 
modifying the 3D animation information according to 
the modified duration of the clip object. 

2. The method of claim 1, wherein said step of modifying 
65 said at least one animation parameter comprises the steps of 

modifying at least one of a first and a last one of the 
animation frames in said sequence, and re-rendering inter- 
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vening ones of the animation frames in accordance with an 
extent of the frame modification. 

3. Asystem for creating and modifying an edit comprising 
3D animation information, comprising: 

a storage device for storing a plurality of animation 
elements with associated modifiable animation 
parameters, wherein the animation parameters include 
the 3D animation information; 

a computer operatively connected to the storage device to 
access the animation elements stored therein for select- 
ing an animation element from the plurality of anima- 
tion elements; 

means for arranging the selected animation element in an 
associated track so as to form a clip object for graphical 
display relative to a time line; 

means for compositing a plurality of the clip objects so as 
to form a collection of layered tracks defining the edit, 
wherein one of the clip objects includes a sequence of 
2D animation frames associated with the 3D animation 
information and having a start position and an end 
position relative to the time line; 

a tool associated with each animation element for modi- 
fying at least one of the animation parameters to initiate 
independent re-rendering of the animation elements 
associated with the modified animation parameters in 
accordance with an extent of each modification, 
wherein the tool includes means for modifying a dura- 
tion of one of the clip objects by altering one of the 
associated start and end positions of the clip object, and 
means for modifying the 3D animation information 
according to the modified duration of the clip object. 

4. The method of claim 1, wherein each said animation 
parameters are independently modifiable. 

5. The system of claim 1, wherein said step of modifying 
3D animation information includes re-arranging said clip in 
said layered tracks. 

6. The system of claim 1, wherein said animation element 
is a variable angle camera. 

7. The method of claim 1, wherein said animation element 
is the number of light sources. 

8. The method of claim 1, wherein said animation infor- 
mation comprises one of static animation information and 
modifiable animation information. 

9. The system of claim 3, further comprising editing data 
structure for maintaining and co-ordinating the relationship 
between said animation element and said associated tool. 

10. The system of claim 3, wherein said associated tool 
includes one of an animation program, a composite system, 
and a rendering machine. 

11. The system of claim 3, wherein said animation param- 
eters are independently modifiable. 

12. The system of claim 3, wherein said edit is an 
independent clip of a complex animation. 

13. A method for implementing a graphical user interface 
for editing a three-dimensional animation comprising an 
animation element with associated animation parameters, 
wherein the animation parameters define animation of a 
three-dimensional object over time, the method comprising: 

displaying a clip object corresponding to the animation 
element, wherein the clip object is displayed in a track 
in a timeline and has a start position and an end position 
relative to said timeline, wherein the start position and 
the end position define a duration of the clip object, 
wherein the duration of the clip object defines the 
animation parameters associated with the animation 
element; 

rendering the animation element to produce a first ren- 
dered sequence of two dimensional animation frames in 
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accordance with the animation parameters as defined 
by the duration of the clip object; 
compositing the first rendered sequence of two- 
dimensional animation frames with at least one other 
5 clip object representing another sequence of two- 
dimensional frames to produce a representation of the 
three-dimensional animation; 
allowing a user to modify the duration of the clip object 
corresponding to the animation element by manipulat- 
io ing at least one of the start position and the end position 
of the clip object, whereby the animation parameters 
associated with the animation element corresponding to 
the clip object are modified according to the modified 
duration of the clip object; 
is rendering the animation element corresponding to the clip 
object in accordance with the animation parameters as 
defined by any modification to the duration of the clip 
object to produce a second rendered sequence of two 
dimensional animation frames; and 
20 compositing the second rendered sequence of two- 
dimensional animation frames with the at least one 
other clip object representing the other sequence of 
two-dimensional frames to produce a representation of 
the three-dimensional animation. 
25 14. The method of claim 13, wherein the three dimen- 
sional object is defined by information defining a skeleton 
and skin of the three dimensional object in three dimensions 
and the animation is defined by position and motion infor- 
mation of the three dimensional object in three dimensions. 
30 15. The method of claim 13, further comprising modify- 
ing the animation parameters according to the modified 
duration of the clip object such that speed of the three 
dimensional object in the animation is modified. 

16. The method of claim 13, further comprising modify- 
35 ing the animation parameters according to the modified 

duration of the clip object such that a position of the three 
dimensional object in the animation is modified. 

17. The method of claim 1, wherein the animation param- 
eters include information defining a skeleton and skin of a 
three dimensional object in three dimensions and position 

40 information of the three dimensional object in three dimen- 
sions over time. 

18. The method of claim. 1, wherein modifying the 3D 
animation information comprises modifying the animation 
parameters according to the modified duration of the clip 

45 object such that speed of motion of the three dimensional 
object is modified. 

19. The method of claim 1, wherein modifying the 3D 
animation information comprises modifying the animation 
parameters according to the modified duration of the clip 

50 object such that a position of the three dimensional object is 
modified. 

20. The system of claim 3, wherein the animation param- 
eters include information defining a skeleton and skin of a 
three dimensional object in three dimensions and position 

55 information of the three dimensional object in three dimen- 
sions over time. 

21. The system of claim 3, wherein the means for modi- 
fying the 3D animation information comprises means for 
modifying the animation parameters according to the modi- 
fied duration of the clip object such that speed of motion of 

60 the three dimensional object is modified. 

22. The system of claim 3, wherein the means for modi- 
fying the 3D animation information comprises means for 
modifying the animation parameters according to the modi- 
fied duration of the clip object such that a position of the 

65 three dimensional object is modified. 
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